Real-time bite articulation

ABSTRACT

A method may include obtaining a first 3D model of an upper jaw of a patient using an intraoral scanner and obtaining a second 3D model of the lower jaw of the patient using the intraoral scanner. The method may also include capturing a series of 2D images of the upper and lower jaws of the patient as the patient is moves the upper jaw and lower jaw in dynamic occlusion and processing the captured series of 2D images to identify features associated with the upper jaw of the patient and the lower jaw of the patient. For each 2D image in the captured series of 2D images, the method may include identifying a relative position of the first 3D model and the second 3D model based on alignment of features in the first 3D model and second 3D model with the features identified in the 2D image to generate a series of relative positions of the first 3D model and the second 3D model. The method may also include modeling dynamic occlusion of the upper jaw and the lower jaw of the patient based on the series of relative positions of the first 3D model and the second 3D model.

RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Patent Application No. 63/370,065, filed Aug. 1, 2022, andtitled “REAL-TIME BITE ARTICULATION WITH 2D IMAGES,” which isincorporated, in its entirety, by this reference.

BACKGROUND

Creating a virtual bite articulation model using three-dimensionalmodels of a patient's dentition generated with intraoral scanners isless than ideal for a number of reasons. During the process of digitallyaligning an upper jaw with a lower jaw, a 3D scan of a small portion ofa patient's jaws in occlusion may be used to determine the patient'sbite in occlusion. However, such practices result in inaccuracies thatnegatively impact the quality of the determined bite and associatedarticulation. For example, patients often bite their teeth duringscanning operations in a way that is unnatural and not in their normalbite position. The intraoral scanner may also interfere with thepatient's natural and normal bite by imparting forces on the cheek,jaws, and/or mouth. The scanning of a small portion of the patient'sjaws may also result in a lack of scan data for accurately aligning theupper and lower jaws. Patients often also bite with forces high enoughto cause their teeth to move from their open bite position which mayalso result in difficulty and accurately aligning the upper and lowerjaws.

Scans of the patient's jaw may also include errors. The scanning of onlya small portion of the jaws also may not account for scan errors, suchas accumulated scan errors when building 3D models of the patient's jaw.For example, the absolute position of teeth on the right side of the jawand the left side of the jaw may be different due to accumulated scanerror during the scanning process. Such accumulated errors may approach0.5 mm.

SUMMARY

Accordingly, as will be described in greater detail below, the presentdisclosure describes various systems and methods for generating bitearticulation with a combination of three-dimensional and two-dimensionalimaging techniques. The systems and methods disclosed herein may be usedto generate an accurate real-time bite articulation model of a patient'sdentition.

In addition, the systems and methods described herein may improve thefunctioning of a computing device and related systems by reducingcomputing resources and overhead for acquiring scan data and generatingthree-dimensional bite articulation models of the patient's dentition,thereby improving processing efficiency of the computing device overconventional approaches. These systems and methods may also improve thefield of dental treatment, including prosthodontics and orthodontics, byanalyzing data and carrying out methods that lead to more efficient useof dental resources and more accurate bite articulation models.

INCORPORATION BY REFERENCE

All patents, applications, and publications referred to and identifiedherein are hereby incorporated by reference in their entirety and shallbe considered fully incorporated by reference even though referred toelsewhere in the application.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the features, advantages and principles of thepresent disclosure will be obtained by reference to the followingdetailed description that sets forth illustrative embodiments, and theaccompanying drawings of which:

FIG. 1 shows a flow diagram for generating a dynamic occlusion model ofa patient's dentition, in accordance with some embodiments;

FIG. 2 shows a block diagram of an example system for generating adynamic occlusion model of a patient's dentition, in accordance withsome embodiments;

FIG. 3 shows an example apparatus for affixing registration targets on apatient's dentition, in accordance with some embodiments;

FIG. 4 shows an example apparatus for affixing registration targets on apatient's dentition, in accordance with some embodiments;

FIG. 5 depicts an example apparatus for affixing registration targets ona patient's teeth or gingiva, in accordance with some embodiments;

FIG. 6 shows an example of using anatomic features for the gingiva forfeature identification and registering images, in accordance with someembodiments;

FIG. 7 shows an example of using anatomic features for the gingiva forfeature identification and registering images, in accordance with someembodiments;

FIG. 8 shows an example of multiple perspective imaging for featureidentification and registering images, in accordance with someembodiments;

FIG. 9 shows an example of using dye on a patient's dentition forfeature identification and registering images, in accordance with someembodiments;

FIG. 10 shows an example of using two-dimensional to three-dimensionalprojection of tooth features for feature identification and registeringimages, in accordance with some embodiments;

FIG. 11 shows a flow diagram for using two-dimensional tothree-dimensional projection of tooth features for featureidentification and registering images, in accordance with someembodiments;

FIG. 12 shows a block diagram of an example computing system capable ofimplementing one or more embodiments described and/or illustratedherein, in accordance with some embodiments;

FIG. 13 shows a block diagram of an example computing network capable ofimplementing one or more of the embodiments described and/or illustratedherein, in accordance with some embodiments;

FIG. 14A shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 14B a virtual articular, in accordance with some embodiments;

FIG. 15 shows images captured at various jaw positions, in accordancewith some embodiments;

FIG. 16 shows 3D segmentation of a digital model of the patient'sdetention, in accordance with some embodiments

FIG. 17 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 18 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 19 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 20 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 21 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 22 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 23 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition, in accordance with someembodiments;

FIG. 24 shows a a process for extracting lower jaw motion from closedmouth chewing data, in accordance with some embodiments;

FIG. 25 shows matching of facial features and landmarks, in accordancewith some embodiments;

FIG. 26 shows a process of generating patient specific articulationsimulations, in accordance with some embodiments; and

FIG. 27 shows example occlusal mapping.

DETAILED DESCRIPTION

The following detailed description and figures provide a betterunderstanding of the features and advantages of the inventions describedin the present disclosure in accordance with the embodiments disclosedherein. Although the detailed description and figures include manyspecific embodiments, these are provided by way of example only andshould not be construed as limiting the scope of the inventionsdisclosed herein.

As shown in FIG. 1 , an embodiment of a method 100 for generating adynamic occlusion model of a patient's dentition is shown to includeobtaining a first 3D model of an upper jaw of a patient at block 110,obtaining a second 3D model of a lower jaw of a patient at block 120,capturing a series of 2D images of the upper and lower jaws of thepatient as the patient moves its jaws in dynamic occlusion at block 130,identifying surface features associated with the jaws of the patient atblock 140, generating a series of relative positions of the first 3Dmodel and the second 3D model based on the identified surface featuresin the three-dimensional models and the 2D images at block 150, andmodeling a dynamic occlusion of the upper jaw and the lower jaw of thepatient based on the series of relative positions at block 160.

The process shown in FIG. 1 may be performed by any suitablecomputer-executable code and/or computing system, including thesystem(s) illustrated in FIGS. 2, 12, and 13 . In one example, each ofthe steps of the process 100 shown in FIG. 1 may represent an algorithmwhose structure includes and/or is represented by multiple sub-steps,examples of which will be provided in greater detail below.

At block 110, the method may include obtaining a first 3D model of anupper jaw of a patient. A scanner, such as an intraoral scanner, may beused to generate scan data, such as surface topography data, by scanningthe patient's dentition. The surface topography data can be generated bydirectly scanning the intraoral cavity, a physical model (positive ornegative) of the intraoral cavity, or an impression of the intraoralcavity, using a suitable scanning device (e.g., a handheld scanner,desktop scanner, coordinate measuring machine, etc.). During thescanning process, individual frames or images of the patient's teeth maybe used to generate the first 3D model of the upper jaw the patient. Thefirst 3D model of the upper jaw of the patient may include 3D datarepresenting the surface contours and shape of the patient's dentitionalong with color data representing the color of the patient's anatomyassociated with the surface of the patient's teeth, gums, and other oralanatomy. The scan data may be stitched together to generate a 3D modelof the patient's dentition, such as the upper jaw of the patient. The 3Dmodel of the patient's dentition may include lingual, buccal, andocclusal surfaces of the patient's teeth along with buccal and lingualsurfaces of the patient's gingiva. The scan data may include digitalrepresentations of a patient's teeth. The digital representation, suchas the two-dimensional or three-dimensional models may include surfacetopography data for the patient's intraoral cavity (including teeth,gingival tissues, etc.). The surface topography data can be generated bydirectly scanning the intraoral cavity, using a suitable scanning device(e.g., a handheld scanner).

In some embodiments, the scan data may include near infrared images anddata representing subsurface structures and features of the patient'sdentition or other parts of the oral cavity, such as the gingiva. Nearinfrared illumination can penetrate the surface of the patient's teethand gingiva to illuminate subsurface features for capture by an imagesensor that is sensitive to near infrared wavelengths of light. Thesubsurface data may be aligned with the three-dimensional model of thepatient's teeth during the scanning process. In some embodiments the 3Dmodel may be a volumetric model and the subsurface data may be added atsubsurface locations of the 3D model that correspond to the subsurfacelocations of the features in the physical world.

In some embodiments, obtaining the first 3D model of the upper jaw ofthe patient may include capturing images of features associated with thepatient's dentition. In some embodiments, the features may includenatural features, such as anatomic features of the patient's dentition.In some embodiments, the features may include artificial features, suchas features added to the patient's dentition in order to more clearlyidentify locations associated with the patient's jaw, as discussedherein.

At block 120, the method may include obtaining a second 3D model of alower jaw of a patient. A scanner, such as an intraoral scanner, may beused to generate scan data by scanning the patient's dentition. Duringthe scanning process, individual frames or images of the patient's teethmay be used to generate the first 3D model of the lower jaw the patient.The first 3D model of the lower jaw of the patient may include 3D datarepresenting the surface contours and shape of the patient's dentitionalong with color data representing the color of the patient's anatomyassociated with the surface of the patient's teeth. The scan data may bestitched together to generate a 3D model of the patient's dentition,such as the lower jaw of the patient. The 3D model of the patient'sdentition may include lingual, buccal, and occlusal surfaces of thepatient's teeth along with buccal and lingual surfaces of the patient'sgingiva. The scan data may include digital representations of apatient's teeth. The digital representation, such as the two-dimensionalor three-dimensional models may include surface topography data for thepatient's intraoral cavity (including teeth, gingival tissues, etc.).The surface topography data can be generated by directly scanning theintraoral cavity, using a suitable scanning device (e.g., a handheldscanner).

In some embodiments, the scan data may include near infrared images anddata representing subsurface structures and features of the patient'sdentition. Near infrared illumination can penetrate the surface of thepatient's teeth and gingiva to illuminate subsurface features forcapture by an image sensor that is sensitive to near infraredwavelengths of light. The subsurface data may be aligned with thethree-dimensional model of the patient's teeth during the scanningprocess. In some embodiments, the 3D model may be a volumetric model andthe subsurface data may be added at subsurface locations of the 3D modelthat correspond to the subsurface locations of the features in thephysical world.

In some embodiments, obtaining the first 3D model of the lower jaw ofthe patient may include capturing images of features associated with thepatient's dentition. In some embodiments, the features may includenatural features, such as anatomic features of the patient's dentition.In some embodiments, the features may include artificial features, suchas features added to the patient's dentition in order to more clearlyidentify locations associated with the patient's jaw, as discussedherein.

At block 130, the method may include capturing a series of 2D images ofthe upper and lower jaws of the patient as the patient moves its jaws indynamic occlusion. A scanner, such as an intraoral scanner, may be usedto generate 2D scan data by imaging the patient's dentition. The scannermay be the same scanner used to generate the 3D models of the upper andlower jaw of the patient. In some embodiments, the scanner may be adifferent scanner than the scanner used to generate the 3D models of theupper and lower jaws of the patient. During the scanning process,individual frames or images of the patient's teeth may be captured whilethe patient moves their upper and lower jaws relative to each other. Insome embodiments, the images may capture the patient as they move theirjaws from a normal open occlusion through initial occlusion and to ahard fight occlusion. In some embodiments, the captured series of 2Dimages may include various motions of the jaws while in various statesof occlusion such as while moving their jaws in the posterior-anteriordirection and/or in a side-to-side motion in lateral directions. Suchmovements capture the dynamic aspects of the patient's jaw and aid ingenerating an accurate digital three-dimensional real-time articulationmodel of the patient's upper and lower jaws. The paths of the repeatedmotions may be averaged to determine an average or target trajectory ofthe patient's teeth during dynamic occlusion.

Each frame of 2D scan data generated by the scanner includes features ofboth the upper and lower jaws of the patient. The first 2D scan data mayinclude color and other feature data representing the colors andfeatures of the patient's anatomy associated with the surface of thepatient's teeth. In some embodiments, the individual frames or images ofthe 2D scan data may be stitched together to generate larger images ofthe patient's dentition, including both the upper and lower jaw. The 2Dimages of the patient's dentition may include predominantly images ofthe buccal surfaces of the patient's dentition. In some embodiments, theimages may include a buccal, incisal, and/or the occlusal surfaces ofthe patient's dentition.

In some embodiments, the 2D scan data may include near infrared imagesand data representing subsurface structures and features of thepatient's dentition. Near infrared illumination can penetrate thesurface of the patient's teeth and gingiva and illuminate subsurfacefeatures for capture by an image sensor that is sensitive to nearinfrared wavelengths of light. The subsurface data may be aligned withthe 2D surface images of the patient's dentition.

In some embodiments, 2D images of the patient's dentition may includecapturing images of features associated with the patient's dentition. Insome embodiments, the features may include natural features, such asanatomic features of the patient's dentition. In some embodiments, thefeatures may include artificial features, such as features added to thepatient's dentition in order to more clearly identify locationsassociated with the patient's jaw, as discussed herein.

At block 140, the method may include identifying features associatedwith the jaws of the patient. The method may include identifying thefeatures in the 2D data, the 3D model of the patient's upper jaw, and/orthe 3D model of the patient's lower jaw. The features may be anatomicsurface or subsurface features of the patient's anatomy, as discussedherein. In some embodiments, the features may be artificial featuressuch as features added to the patient's dentition, as discussed herein.In some embodiments the features may be targets adhered to or placed onthe patient's dentition. For example, as shown and described withrespect to FIGS. 3, 4, and 5 . In some embodiments, the features may beanatomic features, such as colors or coloring of the patient's gingiva,blood vessels and arteries visible through the patient's gingiva,particular features of the patient's gingiva, such as the interdentalpapillia. For example, as shown and described with respect to FIGS. 6and 7 . In some embodiments, the features may be artificial featuressuch as dyed portions of the patient's dentition such as a plaque dyeapplied to the patient's dentition, or other artificially coloredportions of the patient's dentition. For example, as shown and describedwith respect to FIG. 9 .

In some embodiments, the features may be surface features of thepatient's teeth and/or gingiva that may be captured from multipledirections and/or determined based on a projection of thetwo-dimensional images onto the three-dimensional model of the patient.For example, such as shown with respect to FIGS. 8 and 10 ,respectively.

In some embodiments, the features may be subsurface features or otherfeatures imaged using near infrared imagery, as discussed herein.

In some embodiments, the 2D data may be captured at high rates such asis much as 30 frames per second, 50 frames per second, or 100 frames persecond or more to oversample the motion of the patient's teeth and jaw.Capturing the movement of the patient's jaw at such a high rate allowsfor simplified tracking of the features of the patient's jaw betweenframes and also allows for a sufficient quantity of 2D data to becaptured within a relatively short period of time. This adds to thepatient's comfort by limiting the amount of time the patient's teeth arescanned.

In some embodiments the motion of the patient's teeth may be repeatedseveral times in a row in order to gather data related to the patient'steeth in similar positions over time. For example, a patient may berequested to do a dynamic bite motion multiple times, to slide theirteeth against each other and a lateral and/or anterior or posteriordirection, or in another manner. In some embodiments, the 2D images maybe captured from different positions during the repeated movement of thepatient's teeth. In this way similar movements are captured fromdifferent angles which may then be combined in order to more accuratelydetermine the location of the patient's teeth during dynamic bitemotion, such as for example, as discussed at blocks 150 and 160.

At block 150, a series of relative positions of the first 3D model andthe second 3D model may be generated based on the identified surfacefeatures in the three-dimensional models and the 2D images. At block150, the method 100 locates features in the 3D model of the patient'supper jaw and in the 3D model of the patient's lower jaw that are alsoin the 2D images of the patient's upper and lower jaw and uses thesecommonly found features to align the upper jaw and the lower jaw in theseries of relative positions. For example, the 3D model of the upper jawmay include one or more of an artificial feature, such as a first targetaffixed to the patient's upper jaw or a stained location of plaque, andanatomical feature, such as blood vessels in the gingiva, an outline ofthe patient's tooth (such as shown and described with respect to FIG. 10), or other features on the upper jaw of the patient's dentition.Similarly, the 3D model of the lower jaw may include one or more of anartificial feature, such as a second target affixed to the patient'slower jaw, or anatomical features such as blood vessels and the lowergingiva in outline of a tooth of the patient's lower jaw, or otherfeatures on the lower jaw of the patient's dentition.

The 2D images, either individually or when stitched together, mayinclude images of the features of both the upper jaw and the lower jawin order to facilitate alignment of the upper jaw with the lower jaw.For example, a single image or a stitched image of the patient's upperand lower jaw and dynamic occlusion may include the first target affixedto the upper jaw and the second target affixed to the lower jaw and mayalso include one or more other identified features discussed hereinassociated with the upper jaw the lower jaw. Using the common featuresfound in the 3D model of the upper jaw and the 2D image along withcommon features found in the 3D model of the lower jaw and the 2D image,a relative position and orientation of the upper jaw with respect to thelower jaw can be determined. This process may be repeated many timesover the series of 2D images in order to generate a series of relativepositions of the first 3D model of the upper jaw with respect to thesecond 3D model of the lower jaw.

In some embodiments, the features may be used for the 2D images and notthe 3D models. For example, the features may be imaged from multiplelocations during the 2D imaging at block 130. Then, at block 140, thelocation of the features, such as targets, may be determined based onthe differences in perspectives in two or more 2D images from two ormore locations. The positions of the targets relative to the teeth mayalso be determined based on the 2D images. In some embodiments, at block160 the 2D images may be used to model the dynamic occlusion with the 3Dmodels of the upper and lower jaw.

At block 160, modeling a dynamic occlusion of the upper jaw and thelower jaw of the patient based on the series of relative positions. Thedynamic occlusion of the upper and lower jaw may then be modeled basedon the positions determined at block 150. The model of the dynamicocclusion may be modified as part of a treatment planning process. Forexample, a dental professional may modify the 3D model of the upper jawor the lower jaw with a crown, bridge, implant, or other prosthetic. Themodified 3D model may then be used in the dynamic occlusion model inorder to determine the effect of the prosthetic on the patient's bite.For example, the dynamic occlusion model may be used to determineundesirable tooth contacts or interference or undesirable guidanceduring dynamic occlusion. In this way, the dynamic occlusion model maybe used in place of, for example, a dental articulator.

As shown in FIG. 2 , a system 200 for generating a dynamic occlusionmodel of a patient's dentition may include one or more modules 202. Inone example, all or a portion of the functionality of modules 202 may beperformed by the system 200 and/or any other suitable computing system.As will be described in greater detail below, one or more of modules 202from FIG. 2 may, when executed by at least one processor 230 of thesystem 200, which may be a computing device, enable the system 200 toprovide for the generation of a digital dynamic articulation model. Forexample, and as will be described in greater detail below, one or moreof modules 202 may cause the system 200 to carry out the steps of amethod according to FIG. 1 .

System 200 generally represents any type or form of computing devicecapable of reading computer-executable instructions and are capable ofstoring and analyzing data. System 200 may be, for example, an intraoralscanner and 3D treatment planning computer or may include a scanner 240,such as an intraoral scanner or be operably connected to the scanner.Additional examples of system 200 include, without limitation, laptops,tablets, desktops, servers, cellular phones, Personal Digital Assistants(PDAs), multimedia players, embedded systems, wearable devices (e.g.,smart watches, smart glasses, etc.), smart vehicles, smart packaging(e.g., active or intelligent packaging), gaming consoles, so-calledInternet-of-Things devices (e.g., smart appliances, etc.), variations orcombinations of one or more of the same, and/or any other suitablecomputing device.

Additional examples of system 200 include, without limitation, securityservers, application servers, web servers, storage servers, and/ordatabase servers configured to run certain software applications and/orprovide various security, web, storage, and/or database services.Although illustrated a single entity in FIG. 2 , computing device 200may include and/or represent a plurality of computing devices that workand/or operate in conjunction with one another.

As illustrated in FIG. 2 , system 200 may include one or more memorydevices, such as memory 240. Memory 240 generally represents any type orform of volatile or non-volatile storage device or medium capable ofstoring data and/or computer-readable instructions. In one example,memory 240 may store, load, and/or maintain one or more of modules 202.Examples of memory 240 include, without limitation, Random Access Memory(RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs),Solid-State Drives (SSDs), optical disk drives, caches, variations orcombinations of one or more of the same, and/or any other suitablestorage memory.

As illustrated in FIG. 2 , system 200 may also include one or morephysical processors, such as physical processor 230. Physical processor230 generally represents any type or form of hardware-implementedprocessing unit capable of interpreting and/or executingcomputer-readable instructions. In one example, physical processor 230may access and/or modify one or more of modules 202 stored in memory240. Additionally or alternatively, physical processor 230 may executeone or more of modules 202. Examples of physical processor 230 include,without limitation, microprocessors, microcontrollers, CentralProcessing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) thatimplement softcore processors, Application-Specific Integrated Circuits(ASICs), portions of one or more of the same, variations or combinationsof one or more of the same, and/or any other suitable physicalprocessor.

As illustrated in FIG. 2 , the system 200 may include a scanner 250. Thescanner 250 may have a probe at distal end of a handheld wand. Thescanner may be a multi-modal scanner that may capture images in nearinfrared, white light, and/or narrower band lights, such as green light,red light, or other monochromatic light. The scanner may also include astructure light scanning system or a confocal scanning system forcapturing and generating a 3D surface model of the patient's dentition.The scanner may include one or more imaging systems for capturing imagesfrom multiple perspectives at a time, such as simultaneously.

The system 200 may include an imaging device 260. The imaging device 260may be a 2D or 3D imaging device that captures still or video images ofthe patient's anatomy such as their face and teeth. A 2D imaging devicemay include a color or RGB camera that captures still images of thepatient's face and dentition. In some embodiments the to the imagingdevice may include a color or RGB camera that captures video of thepatient's face and dentition. In some embodiments, a 3D imaging devicesuch as a freebie scanner may be used to capture death the data of thepatient's face and dentition. In some embodiments, the 3D imaging devicemay be a multi-perspective imaging device that captures data frommultiple perspectives at the same time and then generate a 3D modelbased on the images. In some embodiments, the imaging device may beportable imaging device, such as a camera of a cellphone or smartphone.In the embodiments, disclosed herein, the imaging device 260 may beremote from the system 200 and may transmit 2D or 3D image data, to thesystem for processing accordingly to the methods disclosed herein.

The system 200 may be connected to a network. A network may be anymedium or architecture capable of facilitating communication or datatransfer. In one example, a network may facilitate elements of thesystem 200. The network may facilitate communication or data transferusing wireless and/or wired connections. Examples of a network include,without limitation, an intranet, a Wide Area Network (WAN), a Local AreaNetwork (LAN), a Personal Area Network (PAN), the Internet, Power LineCommunications (PLC), a cellular network (e.g., a Global System forMobile Communications (GSM) network), portions of one or more of thesame, variations or combinations of one or more of the same, and/or anyother suitable network.

Additional elements 220 generally represents any type or form of datathat may be used for designing and fabricating temporary and permanentcrown, as discussed herein.

As will be explained in greater detail below, modules 202 may include a3D scanning module 204, a 2D scanning module 206, feature identificationmodule 208, and dynamic occlusion module 210. Although illustrated asseparate elements, one or more of modules 202 in FIG. 2 may representportions of a single module or application.

In certain embodiments, one or more of modules 202 in FIG. 2 mayrepresent one or more software applications or programs that, whenexecuted by a computing device, may cause a computing device, such assystem 200, and associated hardware to perform one or more tasks. Forexample, and as will be described in greater detail below, one or moreof modules 202 may represent modules stored and configured to run on oneor more computing devices, such as the system 200. One or more ofmodules 202 in FIG. 2 may also represent all or portions of one or morespecial-purpose computers configured to perform one or more tasks.

The 3D scanning module 204 running on system 200 may communicate withthe scanner 240 to generate an intraoral scan of the patient'sdentition. The 3D scanning module 204 may provide a user interface thatis shown on a display, where the user interface enables the dentalpractitioner to interact with a user interface associated with 3Dscanning module 204 through manipulation of graphical elements such asgraphical icons and visual indicators such as buttons, menus, and so on.The 3D scanning module 204 may include a number of modes, such as ascanning mode.

The scan mode allows the dental practitioner to capture images and/orvideo of a dental site of the patient's dentition, such as of lowerarch, upper arch, bite segment, and/or a prepared tooth. The imagesand/or video may be used to generate a virtual 3D model of the dentalsite. While in the scan mode, scanning module 204 may register andstitch together intraoral images from the intraoral scanner 240 andgenerate a virtual 3D model of a dental arch.

The 3D scanning module 204 may carry out the process or processes ofblocks 110 and 120 of method 100. For example, the 3D scanning module204 may generate a first 3D model of an upper jaw of a patient and asecond 3D model of a lower jaw of a patient, as discussed with respectto FIG. 1 .

The 3D scanning module 204 running on system 200 may communicate withthe scanner 250 to generate an intraoral scan of the patient'sdentition. The 3D scanning module 204 may provide a user interface thatis shown on a display, where the user interface enables the dentalpractitioner to interact with a user interface associated with scanningmodule 204 through manipulation of graphical elements such as graphicalicons and visual indicators such as buttons, menus, and so on. The 3Dscanning module 204 may include a number of modes, such as a scanningmode.

The scan mode allows the dental practitioner to capture images and/orvideo of a dental site of the patient's dentition, such as the lower andupper arches in occlusion, including dynamic occlusion, lower arch,upper arch, bite segment, and/or a prepared tooth. The images and/orvideo may be used to generate one or more 2D images of the dental site.While in the scan mode, 3D scanning module 204 may register and stitchtogether intraoral 3D images from the intraoral scanner 240.

The 2D scanning module 206 may carry out the process or processes ofblock 130 of method 100. For example, capturing a series of 2D images ofthe upper and lower jaws of the patient as the patient moves its jaws indynamic occlusion.

The feature identification module 208 may identify features associatedwith the upper and lower jaws of the patient within the 3D and 2D scandata, such as the 3D models and the 2D images. For example, the featureidentification module 208 may carry out the process or processes ofblock 140. In some embodiments, the feature identification module 208may identify features in the 2D data, the 3D model of the patient'supper jaw, and/or the 3D model of the patient's lower jaw. The featureidentification module 208 may identify features that are anatomicalfeatures, such as surface features or subsurface features of thepatient's anatomy, as discussed herein. In some embodiments, the featureidentification module 208 may identify features that are artificialfeatures such as features added to the patient's dentition, as discussedherein. In some embodiments, the feature identification module 208 mayidentify features that are targets adhered to or placed on the patient'sdentition. For example, as shown and described with respect to FIGS. 3,4, and 5 . In some embodiments, the feature identification module 208may identify features that are anatomic features, such as colors orcoloring of the patient's gingiva, blood vessels and arteries visiblethrough the patient's gingiva, particular features of the patient'sgingiva, such as the interdental papillia. For example, as shown anddescribed with respect to FIGS. 6 and 7 . In some embodiments, thefeature identification module 208 may identify features that areartificial features such as dyed portions of the patient's dentitionsuch as a plaque dye applied to the patient's dentition, or otherartificially colored portions of the patient's dentition. For example,as shown and described with respect to FIG. 9 .

In some embodiments, the feature identification module 208 may identifyfeatures that are surface features of the patient's teeth and/or gingivathat may be captured from multiple directions and/or determined based ona projection of the two-dimensional images onto the three-dimensionalmodel of the patient. For example, such as shown with respect to FIGS. 8and 10 , respectively. In some embodiments, the feature identificationmodule 208 may identify features that are subsurface features or otherfeatures imaged using near infrared imagery, as discussed herein. Insome embodiments, the feature identification module 208 may identifyfeatures that are captured at high rates such as is much as 30 framesper second, 50 frames per second for 100 frames per second or more tooversample the motion of the patient's teeth and j aw.

The dynamic occlusion module 210 may use the data generated and gatheredby the other modules and additional elements in order to generate adynamic occlusion model and/or to derive an articulator model orarticulator settings for an articulator model of the patient'sdentition, such as described in the methods herein.

As illustrated in FIG. 2 , example system 200 may also include one ormore additional elements 220, such as 3D scan data 224, feature data226, and 2D scan data 228. The 3D scan data 224 may include one or morethree-dimensional models of the patient's anatomy such as their faceincluding their eyes, cheeks, nose, lips, mouth, chin and other facialfeatures and intraoral structure including scans of their dentition,prepared teeth, gingiva, features, and etc. The 3D scan data, alsoreferred to herein as 3D image data and 3D data, may include 3D digitalrepresentations of a patient's anatomy, such as the face and dentition,including the teeth and gingiva and may include point clouds, 3D models,such as 3D surface models, and other 3D representations. The digitalrepresentation, such as three-dimensional models, may include surfacetopography data for the patient's face and intraoral cavity (includingteeth, gingival tissues, features etc.). The surface topography data canbe generated by directly scanning the intraoral cavity, using a suitablescanning device (e.g., scanner 240) or using an extraoral imagingdevice, such as a multiview or other 3D imaging device. In someembodiments, 3D data may be multiview 3D data captured from multipleperspectives using multiple imaging sensors and a fixed or known specialrelationship.

The feature data may be 2D or 3D data representing the features of thepatient's intraoral cavity and face. The feature data may includeprojections of 2D or 3D data, such as 2D data projected on a 3D model ora 3D model projected in two-dimensions. The feature data may includecolor, shape, 3D orientation, and 3D location information related to thefeatures.

The 2D scan data 228 may include one or more two-dimensional images ofthe patient's anatomy such as their face and intraoral structureincluding scans of their dentition, prepared teeth, gingiva, featuresand etc. The 2D scan data may include digital representations of apatient's teeth. The digital representation, such as two-dimensionalimages, may include surface and subsurface image data of the patient'sintraoral cavity (including teeth, gingival tissues, features etc.). Theimage data can be generated by directly scanning the intraoral cavity,using a suitable scanning device (e.g., scanner 240).

Accurately generating a series of relative positions of two 3D modelssuch as the 3D model of the upper jaw and the 3D model of the lower jawusing captured 2D images may use stable features. Stable features arefeatures that remain in the same relative position with respect to anupper jaw or the lower jaw during the scanning process. For example, ifa feature is used in the 3D scanning of the upper jaw and 2D images ofthe upper jaw and lower jaw in dynamic occlusion then the feature shouldremain in the same or very close to the same position relative to theupper jaw during the 3D scanning and 2D imaging process. Similarly, if afeature is used in the 3D scanning of the lower jaw and the 2D imagingof the upper jaw and lower jaw in dynamic occlusion then the featureshould remain in the same or very close to the same position relative tothe lower jaw during the 3D scanning and 2D imaging process. In someembodiments, for example when the features are used only in the 2Dimaging process, and the feature would remain in the same positionrelative to a respective one of the upper jaw and the lower jaw 2Dimaging process.

Many types of artificial and anatomical features may be used to modelthe dynamic occlusion of an upper jaw and a lower jaw. For example, FIG.3 depicts a target mounting system 300 for use in stably mounting aregistration target 308 to a patient's teeth. The system 300 may includea lingual support that may be arcuate in shape and shaped to abut orclamp against a lingual surface of one or more of the patient's teethand a buccal support 304 that may be arcuate in shape and shaped to abutor clamp against a buccal surface of one or more of the patient's teeth.The lingual and buccal supports 302, 304 may be rigid or flexible. Arigid support may maintain its shape during use while a flexible supportmay deform and take on, at least partially, the shape of the buccal orlingual surface of the patient's teeth. The system 300 may also includethird support 310 that is coupled to the buccal support or the lingualsupport with one or more deformable, force supplying, members 306, suchas springs. The springs 306 apply a clamping force between the buccalsupport and the lingual support and the teeth.

The mounting system 300 may also include one or more interproximalextensions 312 that extend between the lingual support the buccalsupport and the third support in order to connect the three and retainthem. In some embodiments, the interproximal extensions limit the travelof the third support and apply a counteracting force against theclamping force imparted by the springs.

A registration target 308 may be coupled to the mounting system 300. Insome embodiments the registration target may be coupled to the buccalsupport or the third support. The registration target 308 may have manyshapes such as a cross or crosshair shape, a round or spherical shape,or other shape. The clamping force against the patient's teeth hold theregistration target 308 in a stable position relative to the arch towhich it is attached during the 3D scanning and/or the 2D imagingprocess.

FIG. 4 depicts a stable target system 400. The stable target system 400may include a target 414 stably coupled to one or more mounting fixtures412, 416. The mounting features 412, 416 may be temporarily coupled tothe teeth 422 or the gingiva 420 of the patient's dentition. In someembodiments, a temporary adhesive may be used to couple the mountingfixtures 412, 416 to the patient's dentition. In some embodiments,suction cups may be used to stably couple the mounting fixtures 412, 416to the patient's dentition. In some embodiments, a combination oftemporary adhesive and suction cups or other suction features may beused to temporarily and stably attach the target 414 relative to thepatient's upper or lower jaw.

FIG. 5 depicts an embodiment of a stable target system 500. The stabletarget system 500 may include a target 514 stably coupled to a mountingfixture 512. The mounting fixture 512 may be temporarily coupled to thegingiva 520 of the patient's dentition approximate the patient's tooth522. The mounting fixture 512 may be a suction cup or other vacuumadhesion device may be temporarily affixed to the patient's gingiva. Insome embodiments the vacuum adhesion device may be coupled to a buccalsurface of the patient's teeth.

In some embodiments, anatomical features may be used. FIG. 6 depicts andimage 600 that includes anatomical features 608, 610 associated with thelower arch or jaw 604 and the upper arch or jaw 602. In the image 600,the teeth 606 of the patient's lower jaw are occluded or blocked by theteeth of the patient's upper jaw 602. In such an embodiment, surfacefeatures of the patient's teeth may not be visible during thetwo-dimensional imaging process and may not be used for aligning theupper jaw with a lower jaw in dynamic occlusion. In addition, thesurface condors of the gingiva, which are visible, are relatively flatand devoid of easily identifiable surface contours that may be used todetermine the location of the patient's lower jaw. The patient's lowerjaw does have visible colored and subsurface features 608 that may beused to determine the location and position of the patient's lower jaw.Anatomical features such as the patient's veins and arteries and bloodvessels may be located in stable positions during the scanning processand may be imaged using the color or monochromatic 2D imaging device andthen used in method 100 for determining the dynamic occlusion of thepatient's upper and lower jaws. Similarly, even though the teeth of thepatient's upper jaw 602 are visible, the anatomical features 610 of theupper jaw may also be used to determine the relative position of theupper jaw with the lower jaw based on the 2D images. In someembodiments, near infrared wavelengths may be used to show the bloodvessels in higher contrast with respect to the gingiva and othersurrounding tissue.

The patient's dentition may include other anatomical features visible intwo-dimensional imaging that may be used for determining the relativelocation of the patient's upper jaw. For example, FIG. 7 depicts animage 700 of the patient's teeth in occlusion and uses the apex of theinterdental papillia between the upper central incisors as an anatomicalfeature for determining the position of the patient's upper jaw. Theapex of the interdental papillia 724 is the intersection of the gingivallines 722 and 720 and the interproximal center line 702 between thepatient's central incisors. In some embodiments, the location ofmultiple apex's of multiple interdental papillia may be used asanatomical features for determining the position of the patient's upperand lower jaws. The location of the interdental papillia may also beeasily identified in the original 3D scans of the patient's upper andlower jaws.

FIG. 8 depicts an embodiment wherein the relative position of thepatient's upper jaw and lower jaw is captured simultaneously frommultiple known camera locations. In some embodiments, multiple cameras820 a, 820 b, 820 c, may simultaneously capture images of the patient'sdentition 810 from known left, right, and center positions andorientations relative to the patient's dentition. By using simultaneousimage capture from multiple cameras, the patient's upper and lower jawsmay be imaged in dynamic occlusion. The multiple viewing angles of thecameras may allow for determination of the three-dimensional location ofthe patient's lower and upper jaws during the dynamic occlusion.

FIG. 9 depicts an image 900 of dyed or stained teeth 910 for use ingenerating stable features. A dye or stain may be applied to thepatient's teeth to provide a contrasting color that may be imaged by anintraoral scanner. The dye or stain may adhere or be absorbed to plaque,caries, or demineralized locations 912 of the patient's teeth. The stainmay come in the form of an oral tablet that is chewed and swished aroundin the patient's mouth. During this process the stain is absorbed by theplaque, caries and/or the demineralized portions of the patient's teeth.Stains may be colored with a dye in the visible light spectrum such as ared, blue, or purple. In some embodiments, stains may be colored withdyes visible in the near infrared or ultraviolet wavelengths of light.In some embodiments the stains may fluoresce when exposed to certainwavelengths of light.

The dyed or stained plaque, caries, or demineralized areas of thepatient's teeth which the die or stain is absorbed or adheres is stableor fixed on the patient's teeth and do not move during the short timeperiod of the 3D and 2D scanning process. In this way, the dyed plaque,caries or demineralized areas may be used as features 912 fordetermining the relative location and orientation of the patient's upperand lower jaws.

With reference to FIGS. 10 and 11 , a method for using less stable orunstable surface features with two-dimensional images of the patient'sdentition for aligning and determining the relative positions of apatient's upper and lower jaw is depicted. Less stable or unstablesurface features may be less stable or unstable for multiple reasons.For example, in some embodiments, less stable surface features mayappear to be in different or difficult to determine spatial positionsbased on the angle at which the features are imaged. A patient's teethwhich may have relatively flat and featureless surfaces may be such lessstable or unstable features. Method 1300 may be used to determine therelative position of a patient's jaw 1210 based on 2D images 1200 of thepatient's jaw. At block 1306 a 3D model of the patient's upper and/orlower jaw 1302 and camera calibration data 1304 of a camera along withan estimate of the scanner's position and orientation 1340 relative tothe patient's jaw may be used to project the 3D jaw surface onto anestimate of the cameras focal plane. At block 1308 the expected toothoutline of the projected 3D model is generated. At block 1320 theoutlines 1212 of teeth in the captured two-dimensional images 1322 arecomputed. The tooth outlines 1212 may be computed using image processingtechniques such as edge detection techniques and/or machine learning orother techniques for determining the shapes and locations of the edgesof the patient's teeth within the captured images. At block 1310 thecomputed expected tooth outline 1212 determined at block 1308 iscompared to the computed tooth outlines 1212 in the images determined atblock 1320. If the difference between the expected positions and theactual positions is under a threshold, such as an average differencebetween tooth centers and/or less than a maximum distance, then thepositions of the upper and lower jaw are determined and the methodproceeds to step 1312 and is finished. If the difference between thelocation of the teeth computed at block 1320 as compared to the expectedtooth outlines computed and 1308 exceeds a threshold then, at block 1330a more accurate jaw position is suggested.

The difference in positions may be an average difference in the expectedcenter location of each of the patient's teeth between the 2D image inthe projected 3D image. In some embodiments, the difference may be a sumof the difference in the expected center location of each of thepatient's teeth. In some embodiments other methods may be used todetermine a difference between the computed tooth outlines from the 2Dimages at block 1302 and the computed expected tooth outlines from block1308.

Any known algorithm for iteratively converging on the solution may beused. For example, an affine transformation variation of the iterativeclosest point algorithm may be used to suggest a new jaw-to-scannerposition 1340 use in a second or next step of the iterative method 1300.

This method 1300 may be repeated for each frame in the series of 2Dimages of the upper and lower jaws of the patient captured at block 130of method 100 to determine the relative positions of the upper 3D jawand lower 3D jaw at block 150 of method 100.

FIG. 14A shows a flow diagram for a method 1400 for generating a dynamicocclusion model of lower jaw dynamics of a patient's dentition from siximages of a patient's dentition with cheek retractors.

At block 1405 intra-oral images of the patient's dentition aregenerated. The images of the patient's intraoral cavity 1512 may becaptured while a patient wears a cheek retractor 1514, such as a rubbercheek retractor. The images may be received from an extraoral imagingdevice, or other dental imaging system, such as imaging device 260. Insome embodiments, the extraoral aiming device may be a camera on aportable device, such as a smartphone. In order to determine thearticulation movements of the patient's dentition, the images mayinclude six images of the patient's dentition, each image being capturedwhile the patient is their lower jaw in different positions. Thedifferent positions may aid in determining the close contactarticulation of the patient's dentition. Dental close contactarticulation refers to the way in which the upper and lower teeth comeinto contact or fit together when the jaws are closed. Achieving properdental close contact articulation is provides for optimal oral function,comfort, and overall oral health for the patient.

When the teeth come together during the closing of the jaws, they shouldmake simultaneous and harmonious contact. Ideally, all the teeth shouldtouch evenly, distributing the biting forces across the dental arches.This even contact allows for efficient chewing, speaking, andswallowing, as well as maintaining the stability and health of the teethand supporting structures.

FIG. 15 shows images captured at various jaw positions. The sixpositions are the natural bite 1510, the lateral right 1520, the lateralleft 1530, retrusion 1540, protrusion 1550, and opened bite 1560. Thenatural bite 1510 of the upper and lower jaw refers to the way in whichthe teeth of the upper and lower dental arches come together when thejaws are in their relaxed, resting position. It represents the habitualor physiologically stable position of the jaws and teeth, also known asthe centric occlusion or the bite of maximum intercuspation.

For the left lateral bite image 1520, the patient moves their lower jawto the left of their upper jaw while in occlusion. Lateral left bite,also known as left lateral occlusion or left lateral excursion, refersto the movement of the lower jaw (mandible) to the left side duringchewing or any sideways motion. It describes the contact and alignmentof the upper and lower teeth when the lower jaw moves laterally ortowards the left side.

During a lateral left bite, the lower teeth on the left side come intocontact with the corresponding upper teeth on the left side. Thiscontact occurs while the opposing teeth on the right side maintain adisocclusion or separation to avoid interference during the lateralmovement.

For the right lateral bite image 1530, the patient moves their lower jawto the right of their upper jaw while in occlusion. Right lateral bite,also known as right lateral occlusion or right lateral excursion, refersto the movement of the lower jaw (mandible) to the right side duringchewing or any sideways motion. It describes the contact and alignmentof the upper and lower teeth when the lower jaw moves laterally ortowards the right side.

During a right lateral bite, the lower teeth on the right side come intocontact with the corresponding upper teeth on the right side. Thiscontact occurs while the opposing teeth on the left side maintain adisocclusion or separation to avoid interference during the lateralmovement.

In the retrusion image 1540 the patient retracts their lower jaw inwardsor towards the neck while the teeth are in occlusion. Retrusionocclusion, also known as retruded contact position (RCP) or centricrelation (CR), refers to the specific position of the mandible when itis in its most retruded or posteriorly positioned relationship to themaxilla (upper jaw).

Retrusion occlusion is often considered a reference point in dentistryand is used for various dental procedures, including the fabrication ofdental restorations, occlusal adjustments, and the evaluation ofocclusion. It is distinct from the habitual bite or centric occlusion,discussed above, and is determined by the anatomical relationship of thetemporomandibular joints, muscles, and teeth.

In the protrusion image 1550 the patient extends their lower jawoutwards or away the neck while the teeth are in occlusion. Protrusionocclusion, also known as protrusive contact position or protrusiveinterocclusal position, refers to the position of the mandible when itis protruded or moved forward from the retruded contact position. Itrepresents the relationship between the upper and lower teeth when thelower jaw is in its most advanced position.

Protrusion occlusion provides information about the functional contactbetween the upper and lower teeth during forward jaw movements.Understanding and evaluating protrusion occlusion is used for diagnosingand treating various dental conditions, including malocclusions,temporomandibular joint disorders (TMDs), and the fabrication of dentalrestorations.

In the open bite image 1560, the patient's lower jaw is open relative totheir upper jaw. An open bite refers to a jaw position in which theremandible is displaced vertically lower to cause separation, such as nocontact, between the upper and lower jaws.

Referring back to FIG. 14A, at block 1410 the teeth in the imagescaptured at block 1405 are segmented to generate segmented tooth data.The teeth may be segmented by the segmentation module 212. Thesegmentation module 212 may segment the teeth using a machine learningmodel trained with tagged or labeled images of teeth. Tooth segmentationof a 2D image may include determining which teeth are in the imageand/or which pixels in the image correspond to each of the teeth. Theshape, area, edge, or outline of the teeth in the image may bedetermined during tooth segmentation. In some embodiments, the gingiva,such as the gingiva line may also be segmented from the image. Thegingiva may be segmented for each tooth and each gingiva segment may beassigned to a corresponding tooth of the patient.

FIG. 16 depicts a segmented model of the dentition 1600 that includes asegmented gingiva line 1604 and segmented teeth 1602.

At block 1415, 2D features are extracted from the segmented tooth data.The 2D features may include the center points of each tooth. The centerpoints may include the centroid of the tooth in the captured image. The2D features may also include the contours or edges of the teeth in theimage.

At block 1425 intra-oral 3D scan data of the patient's dentition isgenerated. The 3D scan data may include a 3D model of the intraoralcavity of the patient and may include the teeth and gingiva of thepatient. The images may be received from a scanner, such as scanner 250and may be carried out by the 3D scanning module 204.

At block 1430 the teeth in the 3D scan data captured at block 1425 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeled3D images or scan data of teeth. Tooth segmentation of a 3D image mayinclude determining which teeth are in the 3D scan data and/or whichlocations in the scan data (such as point cloud points) correspond toeach of the teeth and then generating a 3D model of each individualtooth in the dentition. In some embodiments, the gingiva, such as thegingiva line may also be segmented from the 3D scan data. The gingivamay be segmented for each tooth and each gingiva segment may be assignedto a corresponding tooth of the patient.

In some embodiments, the upper and lower arches of the patient arescanned separating and an upper arch model is built from the upper arch3D scan data and a lower arch model is build from the lower arch 3D scandata. In some embodiments, the teeth of the patient may be scanned whilein occlusion to align the upper and lower arches in occlusion in orderto determine the occlusal relationship, such as tooth contacts, betweenthe upper and lower teeth. At block 1440, bite registration data,including tooth contacts and occlusal distances or an occlusal map (amap for each location on the teeth of the distance between correspondingteeth in occlusion) may be extracted from the intra-oral scan data.

At block 1435 3D features are extracted from the segmented tooth data.The 3D features may include the center points of each tooth. The centerpoints may include the centroid of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The 3D features mayalso include the contours or edges of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The projection may bebased on a virtual camera focal length, field of view, and/or distancebetween the virtual camera and the teeth. In some embodiments, theprojection may be made based on data from the 2D images. For example,the 2D images may include meta data, such as the focal length and focusdistance, which may be used for projecting the 3D image. For example,the virtual camera may have the focal length of the focal length in the2D images and the distance between the virtual camera and the 3D modelmay be the focus distance in the meta data of the 2D images.

At block 1420 the segmented 3D teeth are matched with the segmented 2Dteeth in each of the intra-oral images. In some embodiments, thesegmented teeth are aligned based on the extracted 2D features and theextracted 3D features. Aligning the extracted 3D features with theextracted 2D features may include attempting to align the center pointsof each tooth, the contours of each tooth, the edges of each tooth,and/or the gingiva associated with each tooth in the 3D data with thecorresponding features in the 2D data.

Matching the segmented 3D teeth are matched with the segmented 2D teethin each of the intra-oral images may be performed for an entire arch atonce, such as for the upper arch separately from the lower arch.

Aligning a whole arch may be contrasted with aligning each tooth in thesegmented 3D data with the location of the corresponding tooth in the 2Ddata on a tooth by tooth basis. When aligning on a tooth-by-tooth basis,the relative positions of the teeth in the arch may change. However,when aligning an entire arch the relative positions of the teeth may notchange. For example, when the 2D images are captured close in time withthe 3D scan or otherwise without tooth movement between capturing the 2Dimages and the 3D scan, then an arch may be considered static. Aligningthe data may include finding a best fit of the 3D features with thevisible 2D features.

If the alignment is successful, the process may proceed to block 1450.If the alignment is not successful, then the process may proceed toblock 1445.

At block 1445, a bundle adjustment may be performed on the 3D segmenteddata. Bundle adjustment may include making adjustments to the projectionof the 3D data and/or 3D features on the 2D image plane. The adjustmentsmay include adjusting one or more of the focal length of the virtualcamera, the field of view of the virtual camera, the distance betweenthe virtual camera and the 3D model of the teeth, and/or changes to theoptical or lens distortion of the projection. For example, if it isdetermined that that virtual camera was too close or to far from the 3Dmodel, then the distance may be increased or decreased accordingly.Similarly, if the projection is wide or too narrow, the field of view orthe focal length may be changed.

After the bundle adjustment is completed, the process may proceed toblock 1435 for feature extraction and then to block 1420 for featurematching again. The process may iterate through blocks 1445, 1435, and1420 until the features are matched within an acceptable margin oferror.

At block 1450, the optimized lower jaw positions relative to the upperjaw for each of the six images from block 1420 are saved. These may be3D models of the positions or data that represented the relativepositions of the jaws in 3D space.

At block 1455, the jaw movements between each of the positions of theoptimized lower jaw positions relative to the upper jaw may beinterpolated and adjusted based on contact constraints. For example, tosimulate the movement of the lower jaw from the right to the left orfrom the front to the back, the lower jaw may be incrementally movedfrom left to right or front to back. In each incremental position, the3D models of the teeth of the lower and upper arch are checked to putthem in contact and for any penetration of the models of the teeth ofthe lower jaw into the teeth of the upper jaw. Since the teeth are solidin real life, they cannot penetrate one another. If a penetration isdetected, the lower jaw may be moved away from the upper jaw until theteeth contact without penetration. In some embodiments, a minimal amountof penetration may be allowed, such as less than 0.1 mm of penetration.The interpolation may be between lateral left and neutral bite, laterright and neutral bite, retraction and neutral bite, and protrusion andneutral bite.

At block 1460 the interpolated movement of the upper arch 1403 and lowerarch 1404 may be used in a virtual articular, such as the virtualarticulator 1401 shown in FIG. 14B. The movement of the lower arch 1404relative to the upper arch 1403 within the articular may be used toderive the articular settings output at block 1465. The articularsettings may include the range of motion of the patient's condyle, whichis the surface for articulation with the articular disk of thetemporomandibular joint and play a role in defining the patient's jawmovements. These settings may be used for treatment planning andprogress tracking. For example, a crown or bridge may be placed on thepatient's arches using the 3D models and the derived articular settingsto determine the jaw to jaw tooth contacts. In some embodiments, apatient's teeth may be captured with 2D images during orthodontictreatment to track the patient's treatment progress. The individualteeth in the 3D data may be aligned with the teeth in the 2D progresstracking images to generate a model of the patient's upper and lowerarch during treatment.

The optimized lower jaw position and interpolated lower jaw positionsmay be used to inverse articulation motion in an articular simulation todetermine and extract articulator settings such as Benett angles,condylar angles, condylar positions.

Patient specific neural networks may be trained from generatedarticulator movements with different settings. Once the neural networkis trained, the patient specific trained neural network may be used toestimate the articulator settings. This approach could be extended tonon-patient specific training and registered jaw scans.

The articulator in the simulation may be a constrained non-linear leastsquare problem, this would take into account occlusal collisions. Theparameters optimized through this would correspond to the articulatorsettings.

The process may include generating from the virtual articulator(starting from average values) and the 3d jaw scan, the radial basisfunction subspace. Then generating from the lower jaw dynamics and 3djaw scan the RBF subspace.

Using Radial basis function from the virtual articular and radial basisfunction from the lower jaw dynamics and the 3D jaw space, thearticulator settings are derived.

This model may be used in a virtual articular with the derived settingsfrom block 1465 to perform analysis of the patient's bite duringtreatment.

In some embodiments, at block 1460, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from block 1450 and the interpolation at block 1455. Themodeling may result in one or more occlusal maps for differentarrangements of the upper and lower jaw. The modeling and the resultingocclusal maps may be used during treatment planning. For example, theupper or lower arch of the patient may be modified with a crown andbridge or other prosthetic or the teeth or jaw position may be changedbased on a proposed orthodontic treatment. These modifications andchanges to the upper and lower arch may then be modeled based on theupper and lower jaw transform, taking into account the change incontacts between the upper and lower arches caused by the changed toothpositions or prosthetics.

FIG. 17 shows a flow diagram for a method 1700 for generating a dynamicocclusion model of lower jaw dynamics of a patient's dentition fromvideo of a patient's dentition with cheek retractors.

At block 1705 intra-oral video of the patient's dentition are generated.The video of the patient's intraoral cavity may be captured while apatient wears a cheek retractor, such as a rubber cheek retractor. Thevideo may be received from an extra oral imaging device, or other dentalimaging system, such as imaging device 260. In order to determine thearticulation movements of the patient's dentition, the video may berecorded or otherwise generated as the patient moves their detention inocclusion through the five occlusion positions plus the open bite, asshown and described with reference to FIG. 15 . The movement through thepositions may aid in determining the close contact articulation of thepatient's dentition.

At block 1710 the teeth in the video captured at block 1705 aresegmented to generate segmented tooth data. In some embodiments, theteeth in each frame of the video or in a plurality of frames in thevideo between each of the six positions are extracted. The teeth may besegmented by the segmentation module 212. The segmentation module 212may segment the teeth using a machine learning model trained with taggedor labeled images or video of teeth. Tooth segmentation of a 2D videomay include determining which teeth are in the video and/or which pixelsin the video correspond to each of the teeth. The shape, area, edge, oroutline of the teeth in the video or each frame of the video may bedetermined during tooth segmentation. In some embodiments, the gingiva,such as the gingiva line may also be segmented from the video. Thegingiva may be segmented for each tooth and each gingiva segment may beassigned to a corresponding tooth of the patient.

At block 1715, 2D features are extracted from the segmented tooth data.The 2D features may include the center points of each tooth in eachframe of the video. The center points may include the centroid of thetooth in the captured image. The 2D features may also include thecontours or edges of the teeth in each frame of the video.

At block 1725 intra-oral 3D scan data of the patient's dentition isgenerated. The 3D scan data may include a 3D model of the intraoralcavity of the patient and may include the teeth and gingiva of thepatient. The images may be received from a scanner, such as scanner 250and may be carried out by the 3D scanning module 204.

At block 1730 the teeth in the 3D scan data captured at block 1725 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeled3D images or scan data of teeth. Tooth segmentation of a 3D image mayinclude determining which teeth are in the 3D scan data and/or whichlocations in the scan data (such as point cloud points) correspond toeach of the teeth and then generating a 3D model of each individualtooth in the dentition. In some embodiments, the gingiva, such as thegingiva line may also be segmented from the 3D scan data. The gingivamay be segmented for each tooth and each gingiva segment may be assignedto a corresponding tooth of the patient.

In some embodiments, the upper and lower arches of the patient arescanned separating and an upper arch model is built from the upper arch3D scan data and a lower arch model is build from the lower arch 3D scandata. In some embodiments, the teeth of the patient may be scanned whilein occlusion to align the upper and lower arches in occlusion in orderto determine the occlusal relationship, such as tooth contacts, betweenthe upper and lower teeth. At block 1740, bite registration data,including tooth contacts and occlusal distances or an occlusal map (amap for each location on the teeth of the distance between correspondingteeth in occlusion) may be extracted from the intra-oral scan data.

At block 1735 3D features are extracted from the segmented tooth data.The 3D features may include the center points of each tooth. The centerpoints may include the centroid of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The 3D features mayalso include the contours or edges of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The projection may bebased on a virtual camera focal length, field of view, and/or distancebetween the virtual camera and the teeth. In some embodiments, theprojection may be made based on data from the 2D video. For example, the2D video may include meta data, such as the focal length and focusdistance, which may be used for projecting the 3D image. For example,the virtual camera may have the focal length of the focal length in the2D video and the distance between the virtual camera and the 3D modelmay be the focus distance in the meta data of the 2D video.

At block 1720 the segmented 3D teeth are matched with the segmented 2Dteeth in each frame of the 2D video. In some embodiments, the segmentedteeth are aligned based on the extracted 2D features and the extracted3D features. Aligning the extracted 3D features with the extracted 2Dfeatures may include attempting to align the center points of eachtooth, the contours of each tooth, the edges of each tooth, and/or thegingiva associated with each tooth in the 3D data with the correspondingfeatures in the 2D data.

Matching the segmented 3D teeth are matched with the segmented 2D teethin each of the frames of the 2D video may be performed for an entirearch at once, such as for the upper arch separately from the lower arch.

Aligning a whole arch may be contrasted with aligning each tooth in thesegmented 3D data with the location of the corresponding tooth in the 2Ddata on a tooth by tooth basis. When aligning on a tooth-by-tooth basis,the relative positions of the teeth in the arch may change. However,when aligning an entire arch the relative positions of the teeth may notchange. For example, when the 2D video is captured close in time withthe 3D scan or otherwise without tooth movement between capturing the 2Dvideo and the 3D scan, then an arch may be considered static. Aligningthe data may include finding a best fit of the 3D features with thevisible 2D features.

If the alignment is successful, the process may proceed to block 1750.If the alignment is not successful, then the process may proceed toblock 1745.

At block 1745, a bundle adjustment may be performed on the 3D segmenteddata. Bundle adjustment may include making adjustments to the projectionof the 3D data and/or 3D features on the 2D image plane of the frames ofthe 2D video. The adjustments may include adjusting one or more of thefocal length of the virtual camera, the field of view of the virtualcamera, the distance between the virtual camera and the 3D model of theteeth, and/or changes to the optical or lens distortion of theprojection. For example, if it is determined that that virtual camerawas too close or too far from the 3D model, then the distance may beincreased or decreased accordingly. Similarly, if the projection is wideor too narrow, the field of view or the focal length may be changed.

After the bundle adjustment is completed, the process may proceed toblock 1735 for feature extraction and then to block 1720 for featurematching again. The process may iterate through blocks 1745, 1735, and1720 until the features are matched within an acceptable margin oferror.

At block 1750, the optimized lower jaw positions relative to the upperjaw for each of the frames of the 2D video from block 1720 are saved.These may be 3D models of the positions or data that represented therelative positions of the jaws in 3D space.

Because the video includes images of the jaw as it moves between each ofthe position in occlusion, the process 1700 may not includeinterpolation, such as described at block 1455 of process 1400.

In some embodiments, at block 1760, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from block 1750. The modeling may result in one or moreocclusal maps for different arrangements of the upper and lower jaw. Themodeling and the resulting occlusal maps may be used during treatmentplanning. For example, the upper or lower arch of the patient may bemodified with a crown and bridge or other prosthetic or the teeth or jawposition may be changed based on a proposed orthodontic treatment. Thesemodifications and changes to the upper and lower arch may then bemodeled based on the upper and lower jaw transform, taking into accountthe change in contacts between the upper and lower arches caused by thechanged tooth positions or prosthetics.

At block 1760 the movement of the upper arch and lower arch captured inthe video and the corresponding 3D models of the positions or data thatrepresent the relative positions of the jaws in 3D space may be used ina virtual articular, such as the virtual articulator 1401 shown in FIG.14B. The movement of the lower arch 1404 relative to the upper arch 1403within the articular may be used to derive the articular settings outputat block 1765. The articular settings may include the range of motion ofthe patient's condyle, which is the surface for articulation with thearticular disk of the temporomandibular joint and play a role indefining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D videoduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images or video to generate a model of thepatient's upper and lower arch during treatment. This model may be usedin a virtual articular with the derived settings from block 1765 toperform analysis of the patient's bite during treatment.

FIG. 18 shows a flow diagram for a method 1800 for generating a dynamicocclusion model of lower jaw dynamics of a patient's dentition fromimages of a patient's dentition with cheek retractors taken frommultiple angles at the same time, such as multiple cameras at differentpositions.

At block 1805 intra-oral images of the patient's dentition aregenerated. The images of the patient's intraoral cavity may be capturedwhile a patient wears a cheek retractor 1514, such as a rubber cheekretractor. The images may be received from an extra oral imaging device,or other dental imaging system, such as imaging device 260. In order todetermine the articulation movements of the patient's dentition,multiple images, each from a different camera position or location aretaken of each of the patient's dentition at the same time while thepatient holds the position of their lower jaw in each of multipledifferent positions. The positions may be one or more of the sixpositions shown and described with respect to FIG. 15 . For example,multiple images for different locations may be taken at the same timewhile the patient holds their jaw the a first position, then multipleimages for different locations may be taken at the same time while thepatient holds their jaw the a second position, and so on. The differentpositions may aid in determining the close contact articulation of thepatient's dentition. Dental close contact articulation refers to the wayin which the upper and lower teeth come into contact or fit togetherwhen the jaws are closed. Achieving proper dental close contactarticulation is provides for optimal oral function, comfort, and overalloral health for the patient. In some embodiments, the images for eachjaw position may be captured within less than 500 ms of each other. Insome embodiments, the images for each jaw position may be capturedwithin less than 250 ms or less than 100 ms of each other.

When the teeth come together during the closing of the jaws, they shouldmake simultaneous and harmonious contact. Ideally, all the teeth shouldtouch evenly, distributing the biting forces across the dental arches.This even contact allows for efficient chewing, speaking, andswallowing, as well as maintaining the stability and health of the teethand supporting structures.

At block 1810 the teeth in the images captured at block 1805 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeledimages of teeth. Tooth segmentation of a 2D image may includedetermining which teeth are in the image and/or which pixels in theimage correspond to each of the teeth. The shape, area, edge, or outlineof the teeth in the image may be determined during tooth segmentation.In some embodiments, the gingiva, such as the gingiva line may also besegmented from the image. The gingiva may be segmented for each toothand each gingiva segment may be assigned to a corresponding tooth of thepatient.

At block 1815, 2D features are extracted from the segmented tooth data.The 2D features may include the center points of each tooth. The centerpoints may include the centroid of the tooth in the captured image. The2D features may also include the contours or edges of the teeth in theimage.

At block 1825 intra-oral 3D scan data of the patient's dentition isgenerated. The 3D scan data may include a 3D model of the intraoralcavity of the patient and may include the teeth and gingiva of thepatient. The images may be received from a scanner, such as scanner 250and may be carried out by the 3D scanning module 204.

At block 1830 the teeth in the 3D scan data captured at block 1825 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeled3D images or scan data of teeth. Tooth segmentation of a 3D image mayinclude determining which teeth are in the 3D scan data and/or whichlocations in the scan data (such as point cloud points) correspond toeach of the teeth and then generating a 3D model of each individualtooth in the dentition. In some embodiments, the gingiva, such as thegingiva line may also be segmented from the 3D scan data. The gingivamay be segmented for each tooth and each gingiva segment may be assignedto a corresponding tooth of the patient.

In some embodiments, the upper and lower arches of the patient arescanned separating and an upper arch model is built from the upper arch3D scan data and a lower arch model is build from the lower arch 3D scandata. In some embodiments, the teeth of the patient may be scanned whilein occlusion to align the upper and lower arches in occlusion in orderto determine the occlusal relationship, such as tooth contacts, betweenthe upper and lower teeth. At block 1840, bite registration data,including tooth contacts and occlusal distances or an occlusal map (amap for each location on the teeth of the distance between correspondingteeth in occlusion) may be extracted from the intra-oral scan data.

At block 1835 3D features are extracted from the segmented tooth data.The 3D features may include the center points of each tooth. The centerpoints may include the centroid of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The 3D features mayalso include the contours or edges of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The projection may bebased on a virtual camera focal length, field of view, and/or distancebetween the virtual camera and the teeth. In some embodiments, theprojection may be made based on data from the 2D images. For example,the 2D images may include meta data, such as the focal length and focusdistance, which may be used for projecting the 3D image. For example,the virtual camera may have the focal length of the focal length in the2D images and the distance between the virtual camera and the 3D modelmay be the focus distance in the meta data of the 2D images.

At block 1820 the segmented 3D teeth are matched with the segmented 2Dteeth in each of the intra-oral images. In some embodiments, thesegmented teeth are aligned based on the extracted 2D features and theextracted 3D features. Aligning the extracted 3D features with theextracted 2D features may include attempting to align the center pointsof each tooth, the contours of each tooth, the edges of each tooth,and/or the gingiva associated with each tooth in the 3D data with thecorresponding features in the 2D data.

Matching the segmented 3D teeth are matched with the segmented 2D teethin each of the intra-oral images may be performed for an entire arch atonce, such as for the upper arch separately from the lower arch.

Aligning a whole arch may be contrasted with aligning each tooth in thesegmented 3D data with the location of the corresponding tooth in the 2Ddata on a tooth by tooth basis. When aligning on a tooth-by-tooth basis,the relative positions of the teeth in the arch may change. However,when aligning an entire arch the relative positions of the teeth may notchange. For example, when the 2D images are captured close in time withthe 3D scan or otherwise without tooth movement between capturing the 2Dimages and the 3D scan, then an arch may be considered static. Aligningthe data may include finding a best fit of the 3D features with thevisible 2D features.

If the alignment is successful, the process may proceed to block 1850.If the alignment is not successful, then the process may proceed toblock 1845.

At block 1845, a bundle adjustment may be performed on the 3D segmenteddata. Bundle adjustment may include making adjustments to the projectionof the 3D data and/or 3D features on the 2D image plane. The adjustmentsmay include adjusting one or more of the focal length of the virtualcamera, the field of view of the virtual camera, the distance betweenthe virtual camera and the 3D model of the teeth, and/or changes to theoptical or lens distortion of the projection. For example, if it isdetermined that that virtual camera was too close or to far from the 3Dmodel, then the distance may be increased or decreased accordingly.Similarly, if the projection is wide or too narrow, the field of view orthe focal length may be changed.

After the bundle adjustment is completed, the process may proceed toblock 1835 for feature extraction and then to block 1820 for featurematching again. The process may iterate through blocks 1845, 1835, and1820 until the features are matched within an acceptable margin oferror.

In some embodiments, the 2D features of the teeth may be combined and/ortriangulated using each of the views for each jaw position. For example,the center points of the teeth, may be determined for each of the viewsfor each jaw position and then the location in space may be triangulatedbased on the different camera angles from the different imaging devices.Similarly, tooth edges or contours and the gingival edges or contoursgingival edges or contours may be triangulated. In some embodiments,different parts of the gingival edges or contours and the tooth edgesand contours may be visible from different camera positions. The datarelated to the edges and contours may be used combined to form orecomplete edges or contours. The data may also be triangulated todetermine the location in space of the edges and contours.

At block 1850, the optimized lower jaw positions relative to the upperjaw for each of the six jaw positions and views from block 1820 aresaved. These may be 3D models of the positions or data that representedthe relative positions of the jaws in 3D space.

At block 1855, the jaw movements between each of the positions of theoptimized lower jaw positions relative to the upper jaw may beinterpolated and adjusted based on contact constraints. For example, tosimulate the movement of the lower jaw from the right to the left orfrom the front to the back, the lower jaw may be incrementally movedfrom left to right or front to back. In each incremental position, the3D models of the teeth of the lower and upper arch are checked to putthem in contact and for any penetration of the models of the teeth ofthe lower jaw into the teeth of the upper jaw. Since the teeth are solidin real life, they cannot penetrate one another. If a penetration isdetected, the lower jaw may be moved away from the upper jaw until theteeth contact without penetration. In some embodiments, a minimal amountof penetration may be allowed, such as less than 0.1 mm of penetration.The interpolation may be between lateral left and neutral bite, laterright and neutral bite, retraction and neutral bite, and protrusion andneutral bite.

At block 1860 the interpolated movement of the upper arch and lower archmay be used in a virtual articular, such as the virtual articulatorshown in FIG. 14B. The movement of the lower arch relative to the upperarch within the articular may be used to derive the articular settingsoutput at block 1865. The articular settings may include the range ofmotion of the patient's condyle, which is the surface for articulationwith the articular disk of the temporomandibular joint and play a rolein defining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D imagesduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images to generate a model of the patient's upperand lower arch during treatment. This model may be used in a virtualarticular with the derived settings from block 1865 to perform analysisof the patient's bite during treatment.

In some embodiments, at block 1860, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from block 1850 and the interpolations at block 1855. Themodeling may result in one or more occlusal maps for differentarrangements of the upper and lower jaw. The modeling and the resultingocclusal maps may be used during treatment planning. For example, theupper or lower arch of the patient may be modified with a crown andbridge or other prosthetic or the teeth or jaw position may be changedbased on a proposed orthodontic treatment. These modifications andchanges to the upper and lower arch may then be modeled based on theupper and lower jaw transform, taking into account the change incontacts between the upper and lower arches caused by the changed toothpositions or prosthetics.

FIG. 19 shows a flow diagram for a method 1900 for generating a dynamicocclusion model of lower jaw dynamics of a patient's dentition fromimages of a patient's dentition with cheek retractors taken frommultiple angles, such as multiple camera positions, at different times,such as more than 1 second apart and may be taken with the same camera.

At block 1905 intra-oral images of the patient's dentition aregenerated. The images of the patient's intraoral cavity may be capturedwhile a patient wears a cheek retractor 1514, such as a rubber cheekretractor. The images may be received from an extra oral imaging device,or other dental imaging system, such as imaging device 260. In order todetermine the articulation movements of the patient's dentition,multiple images, each from a different camera position or location aretaken of each of the patient's dentition while the patient holds theposition of their lower jaw in different positions. The positions may beone or more of the six positions shown and described with respect toFIG. 15 . The different positions may aid in determining the closecontact articulation of the patient's dentition. Dental close contactarticulation refers to the way in which the upper and lower teeth comeinto contact or fit together when the jaws are closed. Achieving properdental close contact articulation is provides for optimal oral function,comfort, and overall oral health for the patient.

At block 1910 the teeth in the images captured at block 1905 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeledimages of teeth. Tooth segmentation of a 2D image may includedetermining which teeth are in the image and/or which pixels in theimage correspond to each of the teeth. The shape, area, edge, or outlineof the teeth in the image may be determined during tooth segmentation.In some embodiments, the gingiva, such as the gingiva line may also besegmented from the image. The gingiva may be segmented for each toothand each gingiva segment may be assigned to a corresponding tooth of thepatient.

At block 1915, 2D features are extracted from the segmented tooth data.The 2D features may include the center points of each tooth. The centerpoints may include the centroid of the tooth in the captured image. The2D features may also include the contours or edges of the teeth in theimage.

At block 1912 bundle adjustment is carried out for the images capturedat block 1905 to adjust the images for variations in focal length,distance between camera and teeth, etc. In some embodiments, bundleadjustment includes determining the focal length, the distance betweenthe camera and the teeth, lens distortions, and other properties ofimage capture system used to capture the image sequence in 1905.

At block 1914 the camera poses, including camera positions aredetermined based on the bundle adjustment.

At block 1925 intra-oral 3D scan data of the patient's dentition isgenerated. The 3D scan data may include a 3D model of the intraoralcavity of the patient and may include the teeth and gingiva of thepatient. The images may be received from a scanner, such as scanner 250and may be carried out by the 3D scanning module 204.

At block 1930 the teeth in the 3D scan data captured at block 1925 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeled3D images or scan data of teeth. Tooth segmentation of a 3D image mayinclude determining which teeth are in the 3D scan data and/or whichlocations in the scan data (such as point cloud points) correspond toeach of the teeth and then generating a 3D model of each individualtooth in the dentition. In some embodiments, the gingiva, such as thegingiva line may also be segmented from the 3D scan data. The gingivamay be segmented for each tooth and each gingiva segment may be assignedto a corresponding tooth of the patient.

In some embodiments, the upper and lower arches of the patient arescanned separating and an upper arch model is built from the upper arch3D scan data and a lower arch model is build from the lower arch 3D scandata. In some embodiments, the teeth of the patient may be scanned whilein occlusion to align the upper and lower arches in occlusion in orderto determine the occlusal relationship, such as tooth contacts, betweenthe upper and lower teeth. At block 1940, bite registration data,including tooth contacts and occlusal distances or an occlusal map (amap for each location on the teeth of the distance between correspondingteeth in occlusion) may be extracted from the intra-oral scan data.

At block 1935 3D features are extracted from the segmented tooth data.The 3D features may include the center points of each tooth. The centerpoints may include the centroid of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The 3D features mayalso include the contours or edges of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The projection may bebased on a virtual camera focal length, field of view, and/or distancebetween the virtual camera and the teeth and other properties of theimage capture system. In some embodiments, the projection may be madebased on data from the 2D images. The process at block 1935 may use thecamera poses determined at block 1914 to generate a 2D projection forextracting the 3D features. In some embodiments, the 2D images mayinclude meta data, such as the focal length and focus distance, whichmay be used for projecting the 3D image. For example, the virtual cameramay have the focal length of the focal length in the 2D images and thedistance between the virtual camera and the 3D model may be the focusdistance in the meta data of the 2D images.

At block 1920 the segmented 3D teeth are matched with the segmented 2Dteeth in each of the intra-oral images for each set of images, a setincluding multiple camera views of a jaw position. In some embodiments,the segmented teeth are aligned based on the extracted 2D features andthe extracted 3D features. Aligning the extracted 3D features with theextracted 2D features may include attempting to align the center pointsof each tooth, the contours of each tooth, the edges of each tooth,and/or the gingiva associated with each tooth in the 3D data with thecorresponding features in the 2D data.

Matching the segmented 3D teeth are matched with the segmented 2D teethin each of the intra-oral images may be performed for an entire arch atonce, such as for the upper arch separately from the lower arch.

Aligning a whole arch may be contrasted with aligning each tooth in thesegmented 3D data with the location of the corresponding tooth in the 2Ddata on a tooth by tooth basis. When aligning on a tooth-by-tooth basis,the relative positions of the teeth in the arch may change. However,when aligning an entire arch the relative positions of the teeth may notchange. For example, when the 2D images are captured close in time withthe 3D scan or otherwise without tooth movement between capturing the 2Dimages and the 3D scan, then an arch may be considered static. Aligningthe data may include finding a best fit of the 3D features with thevisible 2D features.

At block 1922 the upper arch position from each of the multi-view imagessets are aligned. Since the lower jaw moves relative to the upper jaw,aligning the upper jaw in the images sets the upper jaw position. Then,the differences in the lower jaw between the images provides theocclusion positions.

At block 1950, the optimized lower jaw positions relative to the upperjaw for each of the six images from block 1922 are saved. These may be3D models of the positions or data that represented the relativepositions of the jaws in 3D space.

At block 1955, the jaw movements between each of the positions of theoptimized lower jaw positions relative to the upper jaw may beinterpolated and adjusted based on contact constraints. For example, tosimulate the movement of the lower jaw from the right to the left orfrom the front to the back, the lower jaw may be incrementally movedfrom left to right or front to back. In each incremental position, the3D models of the teeth of the lower and upper arch are checked to putthem in contact and for any penetration of the models of the teeth ofthe lower jaw into the teeth of the upper jaw. Since the teeth are solidin real life, they cannot penetrate one another. If a penetration isdetected, the lower jaw may be moved away from the upper jaw until theteeth contact without penetration. In some embodiments, a minimal amountof penetration may be allowed, such as less than 0.1 mm of penetration.The interpolation may be between lateral left and neutral bite, laterright and neutral bite, retraction and neutral bite, and protrusion andneutral bite.

In some embodiments, at block 1960, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from block 1950 and the interpolations at block 1955. Themodeling may result in one or more occlusal maps for differentarrangements of the upper and lower jaw. The modeling and the resultingocclusal maps may be used during treatment planning. For example, theupper or lower arch of the patient may be modified with a crown andbridge or other prosthetic or the teeth or jaw position may be changedbased on a proposed orthodontic treatment. These modifications andchanges to the upper and lower arch may then be modeled based on theupper and lower jaw transform, taking into account the change incontacts between the upper and lower arches caused by the changed toothpositions or prosthetics.

At block 1960 the interpolated movement of the upper arch and lower archmay be used in a virtual articular, such as the virtual articulatorshown in FIG. 14B. The movement of the lower arch relative to the upperarch within the articular may be used to derive the articular settingsoutput at block 1965. The articular settings may include the range ofmotion of the patient's condyle, which is the surface for articulationwith the articular disk of the temporomandibular joint and play a rolein defining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D imagesduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images to generate a model of the patient's upperand lower arch during treatment. This model may be used in a virtualarticular with the derived settings from block 1965 to perform analysisof the patient's bite during treatment.

FIG. 20 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition. At block 2005 multipleintra-oral videos of the patient's dentition are generated in asynchronized manner. Each frame from each of the imaging devices used tocapture the videos may be captured at the same time. In some embodimentsthe frames of the videos may be time stamped and the frames from eachvideo may be synchronized based on their times stamp, such that a framefrom a first of the videos may be synchronized with the frame from eachof the other videos with the closest time stamp. The videos of thepatient's intraoral cavity may be captured while a patient wears a cheekretractor, such as a rubber cheek retractor. The videos may be receivedfrom one or more extra oral imaging devices having one or more imagesensors, or other dental imaging system, such as imaging device 260. Inorder to determine the articulation movements of the patient'sdentition, the video may be recorded or otherwise generated as thepatient moves their detention in occlusion through the five occlusionpositions plus the open bite, as shown and described with reference toFIG. 15 . The movement through the positions may aid in determining theclose contact articulation of the patient's dentition.

At block 2010 the teeth in the videos captured at block 2005 aresegmented to generate segmented tooth data. In some embodiments, theteeth in each frame of the videos or in a plurality of frames in thevideos between each of the six positions are extracted. The teeth may besegmented by the segmentation module 212. The segmentation module 212may segment the teeth using a machine learning model trained with taggedor labeled images or video of teeth. Tooth segmentation of the 2D videosmay include determining which teeth are in the video and/or which pixelsin the video correspond to each of the teeth. The shape, area, edge, oroutline of the teeth in the video or each frame of the video may bedetermined during tooth segmentation. In some embodiments, the gingiva,such as the gingiva line may also be segmented from the video. Thegingiva may be segmented for each tooth and each gingiva segment may beassigned to a corresponding tooth of the patient.

At block 2015, 2D features are extracted from the segmented tooth data.The 2D features may include the center points of each tooth in eachframe of the videos. The center points may include the centroid of thetooth in the captured videos. The 2D features may also include thecontours or edges of the teeth in each frame of the videos.

At block 2025 intra-oral 3D scan data of the patient's dentition isgenerated. The 3D scan data may include a 3D model of the intraoralcavity of the patient and may include the teeth and gingiva of thepatient. The images may be received from a scanner, such as scanner 250and may be carried out by the 3D scanning module 204.

At block 2030 the teeth in the 3D scan data captured at block 2025 aresegmented to generate segmented tooth data. The teeth may be segmentedby the segmentation module 212. The segmentation module 212 may segmentthe teeth using a machine learning model trained with tagged or labeled3D images or scan data of teeth. Tooth segmentation of a 3D image mayinclude determining which teeth are in the 3D scan data and/or whichlocations in the scan data (such as point cloud points) correspond toeach of the teeth and then generating a 3D model of each individualtooth in the dentition. In some embodiments, the gingiva, such as thegingiva line may also be segmented from the 3D scan data. The gingivamay be segmented for each tooth and each gingiva segment may be assignedto a corresponding tooth of the patient.

In some embodiments, the upper and lower arches of the patient arescanned separating and an upper arch model is built from the upper arch3D scan data and a lower arch model is build from the lower arch 3D scandata. In some embodiments, the teeth of the patient may be scanned whilein occlusion to align the upper and lower arches in occlusion in orderto determine the occlusal relationship, such as tooth contacts, betweenthe upper and lower teeth. At block 2040, bite registration data,including tooth contacts and occlusal distances or an occlusal map (amap for each location on the teeth of the distance between correspondingteeth in occlusion) may be extracted from the intra-oral scan data.

At block 2035 3D features are extracted from the segmented tooth data.The 3D features may include the center points of each tooth. The centerpoints may include the centroid of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The 3D features mayalso include the contours or edges of each tooth as shown in a 2Dprojection of a 3D model of the segmented teeth. The projection may bebased on a virtual camera focal length, field of view, and/or distancebetween the virtual camera and the teeth. In some embodiments, theprojection may be made based on data from the 2D video. For example, the2D video may include meta data, such as the focal length and focusdistance, which may be used for projecting the 3D image. For example,the virtual camera may have the focal length of the focal length in the2D video and the distance between the virtual camera and the 3D modelmay be the focus distance in the meta data of the 2D video.

At block 2020 the segmented 3D teeth are matched with the segmented 2Dteeth in each frame of the 2D video. In some embodiments, the segmentedteeth are aligned based on the extracted 2D features and the extracted3D features. Aligning the extracted 3D features with the extracted 2Dfeatures may include attempting to align the center points of eachtooth, the contours of each tooth, the edges of each tooth, and/or thegingiva associated with each tooth in the 3D data with the correspondingfeatures in the 2D data.

Matching the segmented 3D teeth are matched with the segmented 2D teethin each of the frames of the 2D video may be performed for an entirearch at once, such as for the upper arch separately from the lower arch.

Aligning a whole arch may be contrasted with aligning each tooth in thesegmented 3D data with the location of the corresponding tooth in the 2Ddata on a tooth by tooth basis. When aligning on a tooth-by-tooth basis,the relative positions of the teeth in the arch may change. However,when aligning an entire arch the relative positions of the teeth may notchange. For example, when the 2D video is captured close in time withthe 3D scan or otherwise without tooth movement between capturing the 2Dvideo and the 3D scan, then an arch may be considered static. Aligningthe data may include finding a best fit of the 3D features with thevisible 2D features.

If the alignment is successful, the process may proceed to block 2050.If the alignment is not successful, then the process may proceed toblock 2045.

At block 2045, a bundle adjustment may be performed on the 3D segmenteddata. Bundle adjustment may include making adjustments to the projectionof the 3D data and/or 3D features on the 2D image plane of the frames ofthe 2D video. The adjustments may include adjusting one or more of thefocal length of the virtual camera, the field of view of the virtualcamera, the distance between the virtual camera and the 3D model of theteeth, and/or changes to the optical or lens distortion of theprojection. For example, if it is determined that that virtual camerawas too close or too far from the 3D model, then the distance may beincreased or decreased accordingly. Similarly, if the projection is wideor too narrow, the field of view or the focal length may be changed.

After the bundle adjustment is completed, the process may proceed toblock 2035 for feature extraction and then to block 2020 for featurematching again. The process may iterate through blocks 2045, 2035, and2020 until the features are matched within an acceptable margin oferror.

In some embodiments, the 2D features of the teeth may be combined and/ortriangulated from each of the synchronized frames of each video. Forexample, the center points of the teeth, may be determined for eachframe of each video and then the location in space may be triangulatedbased on the different camera angles from the different imaging devices.Similarly, tooth edges or contours and the gingival edges or contoursgingival edges or contours may be triangulated. In some embodiments,different parts of the gingival edges or contours and the tooth edgesand contours may be visible from different cameras. The data related tothe edges and contours may be used combined to form ore complete edgesor contours. The data may also be triangulated to determine the locationin space of the edges and contours.

At block 2050, the optimized lower jaw positions relative to the upperjaw for each of the frames of the 2D videos from block 2020 are saved.These may be 3D models of the positions or data that represented therelative positions of the jaws in 3D space.

Because the video includes images of the jaw as it moves between each ofthe position in occlusion, the process 2000 may not includeinterpolation, such as described at block 1455 of process 1400.

At block 2060 the movement of the upper arch and lower arch captured inthe video and the corresponding 3D models of the positions or data thatrepresent the relative positions of the jaws in 3D space may be used ina virtual articular, such as the virtual articulator 1401 shown in FIG.14B. The movement of the lower arch 1404 relative to the upper arch 1403within the articular may be used to derive the articular settings outputat block 2065. The articular settings may include the range of motion ofthe patient's condyle, which is the surface for articulation with thearticular disk of the temporomandibular joint and play a role indefining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D videoduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images or video to generate a model of thepatient's upper and lower arch during treatment. This model may be usedin a virtual articular with the derived settings from block 2065 toperform analysis of the patient's bite during treatment.

In some embodiments, at block 2060, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the optimized lower jawpositions from block 2050. The modeling may result in one or moreocclusal maps for different arrangements of the upper and lower jaw. Themodeling and the resulting occlusal maps may be used during treatmentplanning. For example, the upper or lower arch of the patient may bemodified with a crown and bridge or other prosthetic or the teeth or jawposition may be changed based on a proposed orthodontic treatment. Thesemodifications and changes to the upper and lower arch may then bemodeled based on the upper and lower jaw transform, taking into accountthe change in contacts between the upper and lower arches caused by thechanged tooth positions or prosthetics.

FIG. 21 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition using a 3D face scan and anintraoral scan. At block 2125 intra-oral 3D scan data of the patient'sdentition is generated. The 3D scan data may include a 3D model of theintraoral cavity of the patient and may include the teeth and gingiva ofthe patient. The images may be received from a scanner, such as scanner250 and may be carried out by the 3D scanning module 204.

At block 2104, a 3D face scan of the patient with a closed bite and openlips is captured. At block 2104, an imaging device may capture images ofpatient from multiple angles. The images may be received from anextraoral imaging device, or other dental imaging system, such asimaging device 260. In some embodiments, the extraoral aiming device maybe a camera on a portable device, such as a smartphone. The images canbe a still images/photographs of the head and face of patient or couldbe a video of the head and face of patient. Throughout this disclosure,any reference to a patient's face may also include the head of thepatient as well. The images may be captured by a built-in camera ofmobile device and/or an external media capturing devices coupled(physically or communicatively) to mobile device and sent to andreceived by a system for processing the images.

The images may be transformed into 3D representation of the head andface of patient 2190. The transformation of images and videos into the3D representation may be performed according to any known or to bedeveloped signal and image processing technique for generating 3Drepresentation of objects.

At block 2106, a 3D face scan of the patient with an open bite and openlips is captured. At block 2106, an imaging device may capture images ofpatient from multiple angle. The images may be received from anextraoral imaging device, or other dental imaging system, such asimaging device 260. In some embodiments, the extraoral aiming device maybe a camera on a portable device, such as a smartphone. The images canbe a still images/photographs of the head and face of patient or couldbe a video of the head and face of patient. Throughout this disclosure,any reference to a patient's face may also include the head of thepatient as well. The images may be captured by a built-in camera ofmobile device and/or an external media capturing devices coupled(physically or communicatively) to mobile device.

The images may be transformed into 3D representation of the head andface of patient 2192. The transformation of images and videos into the3D representation may be performed according to any known or to bedeveloped signal and image processing technique for generating 3Drepresentation of objects.

At block 2108 the 3D model of the of the patient's detention from block2125 is registered to the 3D model of the patient's face from block 2104to generate a 3D model of the patient's face and teeth in the correctspatial relationship. The location of the teeth, such as the upper teethand the upper jaw in the 3D model of block 2104 may be used forregistering the 3D model of the teeth, such as the upper teeth and upperjaw from block 2125 in the correct special relationship with the modelof the face. In some embodiments, the 3D model of the of the patient'sdetention from block 2125 is registered to the 3D model of the patient'sface from block 2106 to generate a 3D model of the patient's face andteeth in the correct spatial relationship. The location of the teeth inthe 3D model of block 2106 may be used for registering the 3D model ofthe teeth from block 2125 in the correct special relationship with themodel of the face.

At block 2112, CBCT data may be generated. CBCT data may include surfaceand subsurface 2D and 3D data of the patient's head and face 2194, suchas the location and orientation of roots of the patient's teeth, theshape of the mandible bone, the location and shape of the TMJ and othersubsurface anatomy. The CBCT data may be integrated into the 3D model ofthe patient's face from blocks 2104 and 2106.

At block 2114, facial landmarks including the TMJ and condylar positionare located or marked on the 3D model of the patient's face in one orboth of the open bite and closed bite 3D face scan models.

At block 2116, an estimate of the TMJ position, geometry, and/ormovement constraints is generated based on the 3D face models, CBCTdata, and the marking of the facial landmarks, or any combinationthereof. The estimate of the TMJ position may be used at block 2108 inthe registration process. In some embodiments, the registration includesthe registration of subsurface anatomy including CBCT data. For example,facial features such as the tanrgus, canthus, and other features thatare extracted using a facebow device on a physical patient. The 3Dfeatures are features used to initialize a face bow device, such as thefrankfurt and camper planes. Based on these features, an estimate of thelocation of the left and right condyle of the TMJ in 3D space.

At block 2118 the lower jaw 3D transform is estimated based on the 3Dface registration to 3D intraoral scan data form block 2108. Forexample, After the upper jaw is registered to the face and skull atblock 2108, the lower jaw may be registered to the 3D model with respectto the upper jaw. At block 2118 the 3D model of the of the patient'slower detention from block 2125 is registered to the 3D model of thepatient's face from block 2104 to generate a 3D model of the patient'sface and teeth in the correct spatial relationship. For example, the 3Dmodel with the upper jaw registered to the face at block 2108 may beused with the lower jaw to register the lower jaw to the face. Thelocation of the teeth, such as the lower teeth and the lower jaw in the3D model of block 2104 or the registered model of block 2108 may be usedfor registering the 3D model of the teeth, such as the lower teeth andlower jaw from block 2125 in the correct special relationship with themodel of the face. In some embodiments, the 3D model of the of thepatient's detention from block 2125 is registered to the 3D model of thepatient's face from block 2106 to generate a 3D model of the patient'sface and teeth in the correct spatial relationship. The location of theteeth in the 3D model of block 2106 may be used for registering the 3Dmodel of the teeth from block 2125 in the correct special relationshipwith the model of the face.

The lower jaw 3D transform may include an estimate of the occlusalcontacts and articulation of the lower jaw relative to the upper jaw asthe jaw moves between the six positions discussed herein.

At block 2160 the movement of the upper arch and lower arch from block2118 may be used in a virtual articular, such as the virtual articulatorshown in FIG. 14B. The movement of the lower arch relative to the upperarch within the articular may be used to derive the articular settingsoutput at block 2165. The articular settings may include the range ofmotion of the patient's condyle, which is the surface for articulationwith the articular disk of the temporomandibular joint and play a rolein defining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D imagesduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images to generate a model of the patient's upperand lower arch during treatment. This model may be used in a virtualarticular with the derived settings from block 2165 to perform analysisof the patient's bite during treatment.

In some embodiments, at block 2160, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from blocks 2108 and 2118. The modeling may result in one ormore occlusal maps for different arrangements of the upper and lowerjaw. The modeling and the resulting occlusal maps may be used duringtreatment planning. For example, the upper or lower arch of the patientmay be modified with a crown and bridge or other prosthetic or the teethor jaw position may be changed based on a proposed orthodontictreatment. These modifications and changes to the upper and lower archmay then be modeled based on the upper and lower jaw transform, takinginto account the change in contacts between the upper and lower archescaused by the changed tooth positions or prosthetics.

FIG. 22 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition. At block 2225 intra-oral 3Dscan data of the patient's dentition is generated. The 3D scan data mayinclude a 3D model of the intraoral cavity of the patient and mayinclude the teeth and gingiva of the patient. The images may be receivedfrom a scanner, such as scanner 250 and may be carried out by the 3Dscanning module 204.

At block 2204, a 3D face scan of video the patient with a closed biteand open lips is captured. In some embodiments, the video may includetalking and chewing motions. The images may be received from anextraoral imaging device, or other dental imaging system, such asimaging device 260. In some embodiments, the extraoral aiming device maybe a camera on a portable device, such as a smartphone. At block 2204,an imaging device may capture video of patient from multiple angles. Theimages can be a video of the head and face of patient. The video may becaptured by a built-in camera of mobile device and/or an external mediacapturing devices coupled (physically or wirelessly communicatively) tomobile device.

The video may be transformed into a video 3D representation of the headand face of patient 2190. The transformation of videos into the 3Drepresentation may be performed according to any known or to bedeveloped signal and image processing technique for generating 3Drepresentation of objects.

At block 2217 soft tissue movements from the video of the 3Drepresentation of the head and face of the patient may be used togenerate the soft tissue deformation during jaw articulation. Theprocess may include mapping the movements and/or deformation of multiplepoints on the patient's soft tissue with the movements of the jaw.

At block 2213 a differential simulation based on the CBCT data and thesoft tissue articulation may be generated. The differential simulationderives the lower mandible movements based on the mapped movements ofsoft tissue along with the CBCT data, which includes a model of themandible.

At block 2208 the 3D model of the of the patient's detention from block2225 is registered to the 3D model of the patient's face from block 2104based on the differential simulation 2213 to generate a 3D model of thepatient's face and teeth in the correct spatial relationship. Thelocation of the teeth in the 3D model of block 2104 may be used forregistering the 3D model of the teeth from block 2225 in the correctspecial relationship with the model of the face.

At block 2212, CBCT data may be generated. CBCT data may include surfaceand subsurface 2D and 3D data of the patient's head and face, such asthe location and orientation of roots of the patient's teeth, the shapeof the mandible bone, the location and shape of the TMJ and othersubsurface anatomy. The CBCT data may be integrated into the 3D model ofthe patient's face from block 2204.

At block 2214, facial landmarks including the TMJ and condylar positionare located or marked on the 3D model of the patient's face in one orboth of the open bite and closed bite 3D face scan models.

At block 2216, an estimate of the TMJ position, geometry, and/ormovement constraints is generated based on the 3D face models, CBCTdata, and the marking of the facial landmarks, or any combinationthereof. The estimate of the TMJ position may be used at block 2108 inthe registration process. In some embodiments, the registration includesthe registration of subsurface anatomy including CBCT data. For example,facial features such as the tanrgus, canthus, and other features thatare extracted using a facebow device on a physical patient. The 3Dfeatures are features used to initialize a face bow device, such as thefrankfurt and camper planes. Based on these features, an estimate of thelocation of the left and right condyle of the TMJ in 3D space.

At block 2218 the lower jaw 3D transform is estimated based on the 3Dface registration to 3D intraoral scan data form block 2208. Forexample, After the upper jaw is registered to the face and skull atblock 2108, the lower jaw may be registered to the 3D model with respectto the upper jaw. At block 2118 the 3D model of the of the patient'slower detention from block 2125 is registered to the 3D model of thepatient's face from block 2104 to generate a 3D model of the patient'sface and teeth in the correct spatial relationship. For example, the 3Dmodel with the upper jaw registered to the face at block 2108 may beused with the lower jaw to register the lower jaw to the face. Thelocation of the teeth, such as the lower teeth and the lower jaw in the3D model of block 2104 or the registered model of block 2108 may be usedfor registering the 3D model of the teeth, such as the lower teeth andlower jaw from block 2125 in the correct special relationship with themodel of the face. In some embodiments, the 3D model of the of thepatient's detention from block 2125 is registered to the 3D model of thepatient's face from block 2106 to generate a 3D model of the patient'sface and teeth in the correct spatial relationship. The location of theteeth in the 3D model of block 2106 may be used for registering the 3Dmodel of the teeth from block 2125 in the correct special relationshipwith the model of the face.

The lower jaw 3D transform may include an estimate of the occlusalcontacts and articulation of the lower jaw relative to the upper jaw asthe jaw moves between the six positions discussed herein.

At block 2260 the movement of the upper arch and lower arch from block2218 may be used in a virtual articular, such as the virtual articulatorshown in FIG. 14B. The movement of the lower arch relative to the upperarch within the articular may be used to derive the articular settingsoutput at block 2265. The articular settings may include the range ofmotion of the patient's condyle, which is the surface for articulationwith the articular disk of the temporomandibular joint and play a rolein defining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D imagesduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images to generate a model of the patient's upperand lower arch during treatment. This model may be used in a virtualarticular with the derived settings from block 2265 to perform analysisof the patient's bite during treatment.

In some embodiments, at block 2260, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from blocks 2208 and 2218. The modeling may result in one ormore occlusal maps for different arrangements of the upper and lowerjaw. The modeling and the resulting occlusal maps may be used duringtreatment planning. For example, the upper or lower arch of the patientmay be modified with a crown and bridge or other prosthetic or the teethor jaw position may be changed based on a proposed orthodontictreatment. These modifications and changes to the upper and lower archmay then be modeled based on the upper and lower jaw transform, takinginto account the change in contacts between the upper and lower archescaused by the changed tooth positions or prosthetics.

FIG. 23 shows a flow diagram for a method for generating a dynamicocclusion model of a patient's dentition.

At block 2325 intra-oral 3D scan data of the patient's dentition isgenerated. The 3D scan data may include a 3D model of the intraoralcavity of the patient and may include the teeth and gingiva of thepatient. The images may be received from a scanner, such as scanner 250and may be carried out by the 3D scanning module 204.

At block 2304, a 3D face scan of video the patient with a closed biteand open lips is captured. In some embodiments, the video may includetalking and chewing motions. The images may be received from anextraoral imaging device, or other dental imaging system, such asimaging device 260. In some embodiments, the extraoral aiming device maybe a camera on a portable device, such as a smartphone. At block 2304,an imaging device may capture video of patient from multiple angles. Theimages can be a video of the head and face of patient. The video may becaptured by a built-in camera of mobile device and/or an external mediacapturing devices coupled (physically or communicatively) to mobiledevice and sent to and received by a system for processing the images.

The video may be transformed into a video 3D representation of the headand face of patient 2190. The transformation of videos into the 3Drepresentation may be performed according to any known or to bedeveloped signal and image processing technique for generating 3Drepresentation of objects.

At block 2301 monocular 2D video form a single camera may be captured ofthe patient talking, chewing, and moving between and in the six jawpositions discussed herein. In some embodiments, the video may includetalking and chewing motions. The images may be received from anextraoral imaging device, or other dental imaging system, such asimaging device 260. In some embodiments, the extraoral aiming device maybe a camera on a portable device, such as a smartphone. At block 2304,an imaging device may capture video of patient from multiple angles. Theimages can be a video of the head and face of patient. The video may becaptured by a built-in camera of mobile device and/or an external mediacapturing devices coupled (physically or communicatively) to mobiledevice and sent to and received by a system for processing the images.

At block 2308 the 3D model of the of the patient's detention from block2325 is registered to the 3D model of the patient's face from block 2304to generate a 3D model of the patient's face and teeth in the correctspatial relationship. The location of the teeth in the 3D model of block2104 may be used for registering the 3D model of the teeth from block2225 in the correct special relationship with the model of the face. Theregistration may also use the monocular video and facial landmarks andtheir movement from block 2301 to generate a moveable 3D model of theface and intraoral structure. The landmarks in the video and theirpositions may be used to determine how the model changes with movement.

At block 2312, CBCT data may be generated. CBCT data may include surfaceand subsurface 2D and 3D data of the patient's head and face, such asthe location and orientation of roots of the patient's teeth, the shapeof the mandible bone, the location and shape of the TMJ and othersubsurface anatomy. The CBCT data may be integrated into the 3D model ofthe patient's face from block 2304.

At block 2314, facial landmarks including the TMJ and condylar positionare located or marked on the 3D model of the patient's face in one orboth of the open bite and closed bite 3D face scan models.

At block 2316, an estimate of the TMJ position, geometry, and/ormovement constraints is generated based on the 3D face models, CBCTdata, monocular video, and the marking of the facial landmarks, or anycombination thereof. The estimate of the TMJ position may be used atblock 2108 in the registration process. In some embodiments, theregistration includes the registration of subsurface anatomy includingCBCT data.

At block 2318 the lower jaw 3D transform is estimated based on the 3Dface registration to 3D intraoral scan data form block 2308. The lowerjaw 3D transform is an estimate of the occlusal contacts andarticulation of the lower jaw relative to the upper jaw as the jaw movesbetween the six positions discussed herein.

At block 2360 the movement of the upper arch and lower arch from block2318 may be used in a virtual articular, such as the virtual articulatorshown in FIG. 14B. The movement of the lower arch relative to the upperarch within the articular may be used to derive the articular settingsoutput at block 2365. The articular settings may include the range ofmotion of the patient's condyle, which is the surface for articulationwith the articular disk of the temporomandibular joint and play a rolein defining the patient's jaw movements. These settings may be used fortreatment planning and progress tracking. For example, a crown or bridgemay be placed on the patient's arches using the 3D models and thederived articular settings to determine the jaw to jaw tooth contacts.In some embodiments, a patient's teeth may be captured with 2D imagesduring orthodontic treatment to track the patient's treatment progress.The individual teeth in the 3D data may be aligned with the teeth in the2D progress tracking images to generate a model of the patient's upperand lower arch during treatment. This model may be used in a virtualarticular with the derived settings from block 2365 to perform analysisof the patient's bite during treatment.

In some embodiments, at block 2360, the dynamic occlusion of the upperand lower teeth and jaws may be modeled based on the upper and lower jawtransform from blocks 2308 and 2318. The modeling may result in one ormore occlusal maps for different arrangements of the upper and lowerjaw. The modeling and the resulting occlusal maps may be used duringtreatment planning. For example, the upper or lower arch of the patientmay be modified with a crown and bridge or other prosthetic or the teethor jaw position may be changed based on a proposed orthodontictreatment. These modifications and changes to the upper and lower archmay then be modeled based on the upper and lower jaw transform, takinginto account the change in contacts between the upper and lower archescaused by the changed tooth positions or prosthetics.

FIG. 24 shows a method 2400 of determining lower jaw motion from closedmoth chewing. At block 2402 intraoral images or video, such as describedwith respect to block 1405, 1705, 1805, 1905, and 2015 are captured asdescribed herein. At block 2404 2D and/or 3D face video with closedmouth chewing is captured. In some embodiments, the video may includetalking and chewing motions. The images may be received from anextraoral imaging device, or other dental imaging system, such asimaging device 260. In some embodiments, the extraoral aiming device maybe a camera on a portable device, such as a smartphone. An imagingdevice may capture video of patient from multiple angles. The images canbe a video of the head and face of patient. The video may be captured bya built-in camera of mobile device and/or an external media capturingdevices coupled (physically or communicatively) to mobile device andsent to and received by a system for processing the images.

At block 2406, one of processes 1400, 1700, 1800, 1900, or 2000 arecarried out. The result of these processes may be a digital articulatormodel of the patient's teeth.

At block 2410 processes 2200 or 2300 may be carried out. The result ofthese processes may be additional articular models including thepatient's external features, such as the facial features.

At block 2408 the data from block 2402, the 2D or 3D video with mouthchewing 2404, the results from block 2406, and the results from block2410 are used to match features extracted from the various 2D and 3Ddata and models described herein to generate lower jaw motion based onclosed mouth chewing. The extracted features are stored at block 2412.FIG. 25 depicts the matching of features, such as tooth edges andcontours and tooth centers between the various images and videos withthe patient's jaws in a single position. For example, intra-oralfeatures 2502 are matched between the cheek retractor image and thenon-cheek retractor images with open lips. Extra-oral features 2505,such as soft tissue features that may include lip edges or features,color features on the skin, such so moles, and folds or other softtissue features may be mapped between the closed mouth and open mouthimages. Such matching may occur at multiple lower jaw positions, such asthe six positions describe herein. From this matching, a later video ofthe patient chewing food and talking may be used with the extracted dataof block 2412 to determine the articular and occlusal relationship whenthe patient's mouth is closed and in real-world use without theinterference of a facebow or other device, instead, a video of thepatient having the extracted features may be used.

Other combinations of the processes discussed herein can be performed.FIG. 26 depicts a method 2600 of combining those processes to generate apatient specific articulation model. The outputs from blocks 2602, 2604,and 2608 may be used to generate a patient specific articulation modelat block 2606. For example, based on the outputs from block 2602, 2604,and 2608, where patient specific information regarding 3D Condyle (TMJ)location, 2D Planes (Frankfurt/Camper), 3D registration of Lower/Uppermandible to Skull, 3D physiological movements of lower jaw aredetermined, a full model of dental articulation of a patient can besimulated, including positions and movements between protrusion,retrusion, laterotrusion. The articulation model may be used fortreatment planning, progress tracking and other dental uses. Forexample, the final position of an orthodontic treatment may be used asthe tooth model in the patient specific articulation model to determinehow the patient's jaw may articulate after treatment.

FIG. 27 depicts a visualization of occlusal contacts. The occlusal map2700 may include color coded data that depicts the location and extentof the occlusal contacts at different jaw positions. Occlusal maps maybe generated for each jaw position and the movements between the jawpositions, as described herein based on the articular models andarticulation settings herein. To improve usage of lower jaw dynamicsduring treatment planning, the occlusal contacts during dynamicmandibular motions. The computed values would identify areas with highcontacts. The contacts can be used to visualize the accumulated forceduring the motion by normalizing the force values in the color space.

The occlusal maps can also visualize footprints of the motion on thesurface of the teeth and color code it based on intensity of thecontact.

Computing System

FIG. 12 is a block diagram of an example computing system 1010 capableof implementing one or more of the embodiments described and/orillustrated herein. For example, all or a portion of computing system1010 may perform and/or be a means for performing, either alone or incombination with other elements, one or more of the steps describedherein (such as one or more of the steps illustrated in FIG. 1 ). All ora portion of computing system 1010 may also perform and/or be a meansfor performing any other steps, methods, or processes described and/orillustrated herein.

Computing system 1010 broadly represents any single or multi-processorcomputing device or system capable of executing computer-readableinstructions. Examples of computing system 1010 include, withoutlimitation, workstations, laptops, client-side terminals, servers,distributed computing systems, handheld devices, or any other computingsystem or device. In its most basic configuration, computing system 1010may include at least one processor 1014 and a system memory 1016.

Processor 1014 generally represents any type or form of physicalprocessing unit (e.g., a hardware-implemented central processing unit)capable of processing data or interpreting and executing instructions.In certain embodiments, processor 1014 may receive instructions from asoftware application or module. These instructions may cause processor1014 to perform the functions of one or more of the example embodimentsdescribed and/or illustrated herein.

System memory 1016 generally represents any type or form of volatile ornon-volatile storage device or medium capable of storing data and/orother computer-readable instructions. Examples of system memory 1016include, without limitation, Random Access Memory (RAM), Read OnlyMemory (ROM), flash memory, or any other suitable memory device.Although not required, in certain embodiments computing system 1010 mayinclude both a volatile memory unit (such as, for example, system memory1016) and a non-volatile storage device (such as, for example, primarystorage device 1032, as described in detail below). In one example, oneor more of modules 202 from FIG. 2 may be loaded into system memory1016.

In some examples, system memory 1016 may store and/or load an operatingsystem 1040 for execution by processor 1014. In one example, operatingsystem 1040 may include and/or represent software that manages computerhardware and software resources and/or provides common services tocomputer programs and/or applications on computing system 1010. Examplesof operating system 1040 include, without limitation, LINUX, JUNOS,MICROSOFT WINDOWS, WINDOWS MOBILE, MAC OS, APPLE'S IOS, UNIX, GOOGLECHROME OS, GOOGLE'S ANDROID, SOLARIS, variations of one or more of thesame, and/or any other suitable operating system.

In certain embodiments, example computing system 1010 may also includeone or more components or elements in addition to processor 1014 andsystem memory 1016. For example, as illustrated in FIG. 10 , computingsystem 1010 may include a memory controller 1018, an Input/Output (I/O)controller 1020, and a communication interface 1022, each of which maybe interconnected via a communication infrastructure 1012. Communicationinfrastructure 1012 generally represents any type or form ofinfrastructure capable of facilitating communication between one or morecomponents of a computing device. Examples of communicationinfrastructure 1012 include, without limitation, a communication bus(such as an Industry Standard Architecture (ISA), Peripheral ComponentInterconnect (PCI), PCI Express (PCIe), or similar bus) and a network.

Memory controller 1018 generally represents any type or form of devicecapable of handling memory or data or controlling communication betweenone or more components of computing system 1010. For example, in certainembodiments memory controller 1018 may control communication betweenprocessor 1014, system memory 1016, and I/O controller 1020 viacommunication infrastructure 1012.

I/O controller 1020 generally represents any type or form of modulecapable of coordinating and/or controlling the input and outputfunctions of a computing device. For example, in certain embodiments I/Ocontroller 1020 may control or facilitate transfer of data between oneor more elements of computing system 1010, such as processor 1014,system memory 1016, communication interface 1022, display adapter 1026,input interface 1030, and storage interface 1034.

As illustrated in FIG. 12 , computing system 1010 may also include atleast one display device 1024 coupled to I/O controller 1020 via adisplay adapter 1026. Display device 1024 generally represents any typeor form of device capable of visually displaying information forwardedby display adapter 1026. Similarly, display adapter 1026 generallyrepresents any type or form of device configured to forward graphics,text, and other data from communication infrastructure 1012 (or from aframe buffer, as known in the art) for display on display device 1024.

As illustrated in FIG. 12 , example computing system 1010 may alsoinclude at least one input device 1028 coupled to I/O controller 1020via an input interface 1030. Input device 1028 generally represents anytype or form of input device capable of providing input, either computeror human generated, to example computing system 1010. Examples of inputdevice 1028 include, without limitation, a keyboard, a pointing device,a speech recognition device, variations or combinations of one or moreof the same, and/or any other input device.

Additionally or alternatively, example computing system 1010 may includeadditional I/O devices. For example, example computing system 1010 mayinclude I/O device 1036. In this example, I/O device 1036 may includeand/or represent a user interface that facilitates human interactionwith computing system 1010. Examples of I/O device 1036 include, withoutlimitation, a computer mouse, a keyboard, a monitor, a printer, a modem,a camera, a scanner, a microphone, a touchscreen device, variations orcombinations of one or more of the same, and/or any other I/O device.

Communication interface 1022 broadly represents any type or form ofcommunication device or adapter capable of facilitating communicationbetween example computing system 1010 and one or more additionaldevices. For example, in certain embodiments communication interface1022 may facilitate communication between computing system 1010 and aprivate or public network including additional computing systems.Examples of communication interface 1022 include, without limitation, awired network interface (such as a network interface card), a wirelessnetwork interface (such as a wireless network interface card), a modem,and any other suitable interface. In at least one embodiment,communication interface 1022 may provide a direct connection to a remoteserver via a direct link to a network, such as the Internet.Communication interface 1022 may also indirectly provide such aconnection through, for example, a local area network (such as anEthernet network), a personal area network, a telephone or cablenetwork, a cellular telephone connection, a satellite data connection,or any other suitable connection.

In certain embodiments, communication interface 1022 may also representa host adapter configured to facilitate communication between computingsystem 1010 and one or more additional network or storage devices via anexternal bus or communications channel. Examples of host adaptersinclude, without limitation, Small Computer System Interface (SCSI) hostadapters, Universal Serial Bus (USB) host adapters, Institute ofElectrical and Electronics Engineers (IEEE) 1394 host adapters, AdvancedTechnology Attachment (ATA), Parallel ATA (PATA), Serial ATA (SATA), andExternal SATA (eSATA) host adapters, Fibre Channel interface adapters,Ethernet adapters, or the like. Communication interface 1022 may alsoallow computing system 1010 to engage in distributed or remotecomputing. For example, communication interface 1022 may receiveinstructions from a remote device or send instructions to a remotedevice for execution.

In some examples, system memory 1016 may store and/or load a networkcommunication program 1038 for execution by processor 1014. In oneexample, network communication program 1038 may include and/or representsoftware that enables computing system 1010 to establish a networkconnection 1042 with another computing system (not illustrated in FIG.12 ) and/or communicate with the other computing system by way ofcommunication interface 1022. In this example, network communicationprogram 1038 may direct the flow of outgoing traffic that is sent to theother computing system via network connection 1042. Additionally oralternatively, network communication program 1038 may direct theprocessing of incoming traffic that is received from the other computingsystem via network connection 1042 in connection with processor 1014.

Although not illustrated in this way in FIG. 12 , network communicationprogram 1038 may alternatively be stored and/or loaded in communicationinterface 1022. For example, network communication program 1038 mayinclude and/or represent at least a portion of software and/or firmwarethat is executed by a processor and/or Application Specific IntegratedCircuit (ASIC) incorporated in communication interface 1022.

As illustrated in FIG. 12 , example computing system 1010 may alsoinclude a primary storage device 1032 and a backup storage device 1033coupled to communication infrastructure 1012 via a storage interface1034. Storage devices 1032 and 1033 generally represent any type or formof storage device or medium capable of storing data and/or othercomputer-readable instructions. For example, storage devices 1032 and1033 may be a magnetic disk drive (e.g., a so-called hard drive), asolid state drive, a floppy disk drive, a magnetic tape drive, anoptical disk drive, a flash drive, or the like. Storage interface 1034generally represents any type or form of interface or device fortransferring data between storage devices 1032 and 1033 and othercomponents of computing system 1010. In one example, additional elements220 from FIG. 2 may be stored and/or loaded in primary storage device1032.

In certain embodiments, storage devices 1032 and 1033 may be configuredto read from and/or write to a removable storage unit configured tostore computer software, data, or other computer-readable information.Examples of suitable removable storage units include, withoutlimitation, a floppy disk, a magnetic tape, an optical disk, a flashmemory device, or the like. Storage devices 1032 and 1033 may alsoinclude other similar structures or devices for allowing computersoftware, data, or other computer-readable instructions to be loadedinto computing system 1010. For example, storage devices 1032 and 1033may be configured to read and write software, data, or othercomputer-readable information. Storage devices 1032 and 1033 may also bea part of computing system 1010 or may be a separate device accessedthrough other interface systems.

Many other devices or subsystems may be connected to computing system1010. Conversely, all of the components and devices illustrated in FIG.12 need not be present to practice the embodiments described and/orillustrated herein. The devices and subsystems referenced above may alsobe interconnected in different ways from that shown in FIG. 12 .Computing system 1010 may also employ any number of software, firmware,and/or hardware configurations. For example, one or more of the exampleembodiments disclosed herein may be encoded as a computer program (alsoreferred to as computer software, software applications,computer-readable instructions, or computer control logic) on acomputer-readable medium. The term “computer-readable medium,” as usedherein, generally refers to any form of device, carrier, or mediumcapable of storing or carrying computer-readable instructions. Examplesof computer-readable media include, without limitation,transmission-type media, such as carrier waves, and non-transitory-typemedia, such as magnetic-storage media (e.g., hard disk drives, tapedrives, and floppy disks), optical-storage media (e.g., Compact Disks(CDs), Digital Video Disks (DVDs), and BLU-RAY disks),electronic-storage media (e.g., solid-state drives and flash media), andother distribution systems.

The computer-readable medium containing the computer program may beloaded into computing system 1010. All or a portion of the computerprogram stored on the computer-readable medium may then be stored insystem memory 1016 and/or various portions of storage devices 1032 and1033. When executed by processor 1014, a computer program loaded intocomputing system 1010 may cause processor 1014 to perform and/or be ameans for performing the functions of one or more of the exampleembodiments described and/or illustrated herein. Additionally oralternatively, one or more of the example embodiments described and/orillustrated herein may be implemented in firmware and/or hardware. Forexample, computing system 1010 may be configured as an ApplicationSpecific Integrated Circuit (ASIC) adapted to implement one or more ofthe example embodiments disclosed herein.

FIG. 13 is a block diagram of an example network architecture 1100 inwhich client systems 1110, 1120, and 1130 and servers 1140 and 1145 maybe coupled to a network 1150. As detailed above, all or a portion ofnetwork architecture 1100 may perform and/or be a means for performing,either alone or in combination with other elements, one or more of thesteps disclosed herein (such as one or more of the steps illustrated inFIG. 1 or the other figures herein). All or a portion of networkarchitecture 1100 may also be used to perform and/or be a means forperforming other steps and features set forth in the instant disclosure.

Client systems 1110, 1120, and 1130 generally represent any type or formof computing device or system, such as example computing system 1010 inFIG. 12 . Similarly, servers 1140 and 1145 generally represent computingdevices or systems, such as application servers or database servers,configured to provide various database services and/or run certainsoftware applications. Network 1150 generally represents anytelecommunication or computer network including, for example, anintranet, a WAN, a LAN, a PAN, or the Internet. In one example, clientsystems 1110, 1120, and/or 1130 and/or servers 1140 and/or 1145 mayinclude all or a portion of system 200 from FIG. 2 .

As illustrated in FIG. 13 , one or more storage devices 1160(1)-(N) maybe directly attached to server 1140. Similarly, one or more storagedevices 1170(1)-(N) may be directly attached to server 1145. Storagedevices 1160(1)-(N) and storage devices 1170(1)-(N) generally representany type or form of storage device or medium capable of storing dataand/or other computer-readable instructions. In certain embodiments,storage devices 1160(1)-(N) and storage devices 1170(1)-(N) mayrepresent Network-Attached Storage (NAS) devices configured tocommunicate with servers 1140 and 1145 using various protocols, such asNetwork File System (NFS), Server Message Block (SMB), or CommonInternet File System (CIFS).

Servers 1140 and 1145 may also be connected to a Storage Area Network(SAN) fabric 1180. SAN fabric 1180 generally represents any type or formof computer network or architecture capable of facilitatingcommunication between a plurality of storage devices. SAN fabric 1180may facilitate communication between servers 1140 and 1145 and aplurality of storage devices 1190(1)-(N) and/or an intelligent storagearray 1195. SAN fabric 1180 may also facilitate, via network 1150 andservers 1140 and 1145, communication between client systems 1110, 1120,and 1130 and storage devices 1190(1)-(N) and/or intelligent storagearray 1195 in such a manner that devices 1190(1)-(N) and array 1195appear as locally attached devices to client systems 1110, 1120, and1130. As with storage devices 1160(1)-(N) and storage devices1170(1)-(N), storage devices 1190(1)-(N) and intelligent storage array1195 generally represent any type or form of storage device or mediumcapable of storing data and/or other computer-readable instructions.

In certain embodiments, and with reference to example computing system1010 of FIG. 12 , a communication interface, such as communicationinterface 1022 in FIG. 12 , may be used to provide connectivity betweeneach client system 1110, 1120, and 1130 and network 1150. Client systems1110, 1120, and 1130 may be able to access information on server 1140 or1145 using, for example, a web browser or other client software. Suchsoftware may allow client systems 1110, 1120, and 1130 to access datahosted by server 1140, server 1145, storage devices 1160(1)-(N), storagedevices 1170(1)-(N), storage devices 1190(1)-(N), or intelligent storagearray 1195. Although FIG. 13 depicts the use of a network (such as theInternet) for exchanging data, the embodiments described and/orillustrated herein are not limited to the Internet or any particularnetwork-based environment.

In at least one embodiment, all or a portion of one or more of theexample embodiments disclosed herein may be encoded as a computerprogram and loaded onto and executed by server 1140, server 1145,storage devices 1160(1)-(N), storage devices 1170(1)-(N), storagedevices 1190(1)-(N), intelligent storage array 1195, or any combinationthereof. All or a portion of one or more of the example embodimentsdisclosed herein may also be encoded as a computer program, stored inserver 1140, run by server 1145, and distributed to client systems 1110,1120, and 1130 over network 1150.

As detailed above, computing system 1010 and/or one or more componentsof network architecture 1100 may perform and/or be a means forperforming, either alone or in combination with other elements, one ormore steps of an example method for virtual care.

While the foregoing disclosure sets forth various embodiments usingspecific block diagrams, flowcharts, and examples, each block diagramcomponent, flowchart step, operation, and/or component described and/orillustrated herein may be implemented, individually and/or collectively,using a wide range of hardware, software, or firmware (or anycombination thereof) configurations. In addition, any disclosure ofcomponents contained within other components should be consideredexample in nature since many other architectures can be implemented toachieve the same functionality.

In some examples, all or a portion of example system 200 in FIG. 2 mayrepresent portions of a cloud-computing or network-based environment.Cloud-computing environments may provide various services andapplications via the Internet. These cloud-based services (e.g.,software as a service, platform as a service, infrastructure as aservice, etc.) may be accessible through a web browser or other remoteinterface. Various functions described herein may be provided through aremote desktop environment or any other cloud-based computingenvironment.

In various embodiments, all or a portion of example system 200 in FIG. 2may facilitate multi-tenancy within a cloud-based computing environment.In other words, the software modules described herein may configure acomputing system (e.g., a server) to facilitate multi-tenancy for one ormore of the functions described herein. For example, one or more of thesoftware modules described herein may program a server to enable two ormore clients (e.g., customers) to share an application that is runningon the server. A server programmed in this manner may share anapplication, operating system, processing system, and/or storage systemamong multiple customers (i.e., tenants). One or more of the modulesdescribed herein may also partition data and/or configurationinformation of a multi-tenant application for each customer such thatone customer cannot access data and/or configuration information ofanother customer.

According to various embodiments, all or a portion of example system 200in FIG. 2 may be implemented within a virtual environment. For example,the modules and/or data described herein may reside and/or executewithin a virtual machine. As used herein, the term “virtual machine”generally refers to any operating system environment that is abstractedfrom computing hardware by a virtual machine manager (e.g., ahypervisor). Additionally or alternatively, the modules and/or datadescribed herein may reside and/or execute within a virtualizationlayer. As used herein, the term “virtualization layer” generally refersto any data layer and/or application layer that overlays and/or isabstracted from an operating system environment. A virtualization layermay be managed by a software virtualization solution (e.g., a filesystem filter) that presents the virtualization layer as though it werepart of an underlying base operating system. For example, a softwarevirtualization solution may redirect calls that are initially directedto locations within a base file system and/or registry to locationswithin a virtualization layer.

In some examples, all or a portion of example system 200 in FIG. 2 mayrepresent portions of a mobile computing environment. Mobile computingenvironments may be implemented by a wide range of mobile computingdevices, including mobile phones, tablet computers, e-book readers,personal digital assistants, wearable computing devices (e.g., computingdevices with a head-mounted display, smartwatches, etc.), and the like.In some examples, mobile computing environments may have one or moredistinct features, including, for example, reliance on battery power,presenting only one foreground application at any given time, remotemanagement features, touchscreen features, location and movement data(e.g., provided by Global Positioning Systems, gyroscopes,accelerometers, etc.), restricted platforms that restrict modificationsto system-level configurations and/or that limit the ability ofthird-party software to inspect the behavior of other applications,controls to restrict the installation of applications (e.g., to onlyoriginate from approved application stores), etc. Various functionsdescribed herein may be provided for a mobile computing environmentand/or may interact with a mobile computing environment.

In addition, all or a portion of example system 200 in FIG. 2 mayrepresent portions of, interact with, consume data produced by, and/orproduce data consumed by one or more systems for information management.As used herein, the term “information management” may refer to theprotection, organization, and/or storage of data. Examples of systemsfor information management may include, without limitation, storagesystems, backup systems, archival systems, replication systems, highavailability systems, data search systems, virtualization systems, andthe like.

In some embodiments, all or a portion of example system 200 in FIG. 2may represent portions of, produce data protected by, and/or communicatewith one or more systems for information security. As used herein, theterm “information security” may refer to the control of access toprotected data. Examples of systems for information security mayinclude, without limitation, systems providing managed securityservices, data loss prevention systems, identity authentication systems,access control systems, encryption systems, policy compliance systems,intrusion detection and prevention systems, electronic discoverysystems, and the like.

The process parameters and sequence of steps described and/orillustrated herein are given by way of example only and can be varied asdesired. For example, while the steps illustrated and/or describedherein may be shown or discussed in a particular order, these steps donot necessarily need to be performed in the order illustrated ordiscussed. The various example methods described and/or illustratedherein may also omit one or more of the steps described or illustratedherein or include additional steps in addition to those disclosed.

While various embodiments have been described and/or illustrated hereinin the context of fully functional computing systems, one or more ofthese example embodiments may be distributed as a program product in avariety of forms, regardless of the particular type of computer-readablemedia used to actually carry out the distribution. The embodimentsdisclosed herein may also be implemented using software modules thatperform certain tasks. These software modules may include script, batch,or other executable files that may be stored on a computer-readablestorage medium or in a computing system. In some embodiments, thesesoftware modules may configure a computing system to perform one or moreof the example embodiments disclosed herein.

As described herein, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each comprise atleast one memory device and at least one physical processor.

The term “memory” or “memory device,” as used herein, generallyrepresents any type or form of volatile or non-volatile storage deviceor medium capable of storing data and/or computer-readable instructions.In one example, a memory device may store, load, and/or maintain one ormore of the modules described herein. Examples of memory devicescomprise, without limitation, Random Access Memory (RAM), Read OnlyMemory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives(SSDs), optical disk drives, caches, variations or combinations of oneor more of the same, or any other suitable storage memory.

In addition, the term “processor” or “physical processor,” as usedherein, generally refers to any type or form of hardware-implementedprocessing unit capable of interpreting and/or executingcomputer-readable instructions. In one example, a physical processor mayaccess and/or modify one or more modules stored in the above-describedmemory device. Examples of physical processors comprise, withoutlimitation, microprocessors, microcontrollers, Central Processing Units(CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcoreprocessors, Application-Specific Integrated Circuits (ASICs), portionsof one or more of the same, variations or combinations of one or more ofthe same, or any other suitable physical processor.

Although illustrated as separate elements, the method steps describedand/or illustrated herein may represent portions of a singleapplication. In addition, in some embodiments one or more of these stepsmay represent or correspond to one or more software applications orprograms that, when executed by a computing device, may cause thecomputing device to perform one or more tasks, such as the method step.

In addition, one or more of the devices described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. Additionally or alternatively, one or more of themodules recited herein may transform a processor, volatile memory,non-volatile memory, and/or any other portion of a physical computingdevice from one form of computing device to another form of computingdevice by executing on the computing device, storing data on thecomputing device, and/or otherwise interacting with the computingdevice.

The term “computer-readable medium,” as used herein, generally refers toany form of device, carrier, or medium capable of storing or carryingcomputer-readable instructions. Examples of computer-readable mediacomprise, without limitation, transmission-type media, such as carrierwaves, and non-transitory-type media, such as magnetic-storage media(e.g., hard disk drives, tape drives, and floppy disks), optical-storagemedia (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), andBLU-RAY disks), electronic-storage media (e.g., solid-state drives andflash media), and other distribution systems.

A person of ordinary skill in the art will recognize that any process ormethod disclosed herein can be modified in many ways. The processparameters and sequence of the steps described and/or illustrated hereinare given by way of example only and can be varied as desired. Forexample, while the steps illustrated and/or described herein may beshown or discussed in a particular order, these steps do not necessarilyneed to be performed in the order illustrated or discussed.

The various exemplary methods described and/or illustrated herein mayalso omit one or more of the steps described or illustrated herein orcomprise additional steps in addition to those disclosed. Further, astep of any method as disclosed herein can be combined with any one ormore steps of any other method as disclosed herein.

The processor as described herein can be configured to perform one ormore steps of any method disclosed herein. Alternatively or incombination, the processor can be configured to combine one or moresteps of one or more methods as disclosed herein.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and shall have the same meaning as theword “comprising.

The processor as disclosed herein can be configured with instructions toperform any one or more steps of any method as disclosed herein.

It will be understood that although the terms “first,” “second,”“third”, etc. may be used herein to describe various layers, elements,components, regions or sections without referring to any particularorder or sequence of events. These terms are merely used to distinguishone layer, element, component, region or section from another layer,element, component, region or section. A first layer, element,component, region or section as described herein could be referred to asa second layer, element, component, region or section without departingfrom the teachings of the present disclosure.

As used herein, the term “or” is used inclusively to refer items in thealternative and in combination.

As used herein, characters such as numerals refer to like elements.

The present disclosure includes the following numbered clauses.

Clause 1. A system for modeling dental articulation of a patient, thesystem comprising: a processor in electronic communication with anextraoral imaging device and an intraoral imaging device; andnon-transitory computer readable medium having instructions storedthereon that when executed by the processor cause the system to: captureextraoral 2D images of the patient's dentition in a plurality ofocclusion positions with the extraoral imaging device; capture a 3Dmodel of the patient's dentition; align the 3D model of the patient'sdentition with the patient's dentition in the extraoral 2D images foreach of the plurality of occlusion positions; and derive digitalarticulator settings for a digital articulator based on the 3D model ofthe patient's dentition with the patient's dentition in the extraoral 2Dimages for each of the plurality of occlusion positions.

Clause 2. The system of clause 1, further comprising instructions tosegment the patient's dentition in the extraoral 2D images.

Clause 3. The system of any one of the preceding clauses, furthercomprising instructions to extract 2D dental features from the segmentedextraoral 2D images of the patient's dentition.

Clause 4. The system of any one of the preceding clauses, wherein the 2Ddental features are one or more of tooth location, tooth outline,gingival line, and tooth centers.

Clause 5. The system of any one of the preceding clauses, furthercomprising instructions to segment the 3D model of the patient'sdentition.

Clause 6. The system of any one of the preceding clauses, furthercomprising instructions to extract 3D dental features from the segmented3D model of the patient's dentition.

Clause 7. The system of any one of the preceding clauses, wherein the 3Ddental features are one or more of tooth location, tooth outline,gingival line, and tooth centers.

Clause 8. The system of any one of the preceding clauses, wherein theinstructions to align the 3D model of the patient's dentition with thepatient's dentition in the extraoral 2D images for each of the pluralityof occlusion positions include instructions to align the 3D dentalfeatures with the 2D dental features.

Clause 9. The system of clause 8, wherein the instructions further causethe system to determine whether the alignment of the 3D dental featureswith the 2D dental features is within a threshold of alignment.

Clause 10. The system of clause 9, wherein the instructions furthercause the system to repeatedly: adjust the 3D features; attempt to alignthe 3D features with the 2D features; and determine whether thealignment of the 3D dental features with the 2D dental features iswithin a threshold of alignment.

Clause 11. The system of clause 10, wherein the instructions to adjustthe 3D features includes instructions to: adjust a 2D projection of the3D features.

Clause 12. The system of clause 11, wherein the instruction to adjust a2D projection of the 3D features includes instructions to: adjust afocal length, virtual camera distance, or lens distortion of the 2Dprojection of the 3D features.

Clause 13. The system of any one of the preceding clauses, furthercomprising instructions to: interpolate jaw movement between theplurality of occlusion positions.

Clause 14. The system of any one of the preceding clauses, wherein theinstructions to interpolate jaw movement between the plurality ofocclusion positions accounts for contact between teeth of the upper archand teeth of the lower arch.

Clause 15. The system of any one of the preceding clauses, wherein theocclusion positions include occlusions positions wherein the lower jawis in a neutral bite, a lateral right bite, a lateral left bite, aretraction bite, and a protrusion bite.

Clause 16. The system of clause 15, wherein the occlusion positionsinclude on open bite.

Clause 17. The system of any one of the preceding clauses, wherein theextraoral 2D images of the patient's dentition are 2D still images ineach of the occlusion positions.

Clause 18. The system of any one of the preceding clauses, wherein theextraoral 2D images of the patient's dentition extraoral 2D images ofthe patient's dentition includes a video as the patient moves theirlower jaw between and to each of the occlusion positions.

Clause 19. The system of any one of the preceding clauses, wherein theextraoral 2D images of the patient's dentition includes 2D still imagesfrom multiple camera angles in each of the occlusion positions.

Clause 20. The system of any one of the preceding clauses, wherein theextraoral 2D images of the patient's dentition includes 2D video frommultiple camera angles as the patient moves their lower jaw between andto each of the occlusion positions.

Clause 21. A system for modeling dental articulation of a patient, thesystem comprising: a processor in electronic communication with anextraoral imaging device and an intraoral imaging device; andnon-transitory computer readable medium having instructions storedthereon that when executed by the processor cause the system to: capture3D data of the patient's face in with the extraoral imaging device;capture a 3D model of the patient's dentition; align the 3D model of thepatient's dentition with 3D data of the patient's face; and generate anestimate of the temporomandibular joint characteristics based on the 3Dmodel of the patient's face; and derive digital articulator settings fora digital articulator based on the 3D model of the patient's dentitionwith the 3D data of the patient's face and the estimate of thetemporomandibular joint characteristics.

Clause 22. The system of clause 21, wherein the 3D data of the patient'sface includes 3D data of the patient's face in with a closed bite andopen lips.

Clause 23. The system of clause 22, wherein the 3D data of the patient'sface includes 3D data of the patient's face in with an open bite andopen lips.

Clause 24. The system of clause 23, wherein the instruction to align the3D model of the patient's dentition with 3D data of the patient's faceincludes aligning registering the 3D model of the patient's dentition tothe 3D data of the patient's face.

Clause 25. The system of any one of clauses 21-24, further comprisinginstructions to generate CBCT data for the internal structure of thepatient's face and jaw.

Clause 26. The system of clause 25, wherein the instructions to generatethe estimate of the temporomandibular joint characteristics based on the3D model of the patient's face further comprise instructions to includeinstructions to generate the estimate of the temporomandibular jointcharacteristics based on the 3D model of the patient's face and the CBCTdata.

Clause 27. The system of clause 21, wherein the instructions includesreceiving markings on the 3D model of the patient's face of thetemporomandibular joint and the condylar position.

Clause 28. The system of clause 27, wherein the instructions to generatethe estimate of the temporomandibular joint characteristics based on the3D model of the patient's face further comprise instructions to includeinstructions to generate the estimate of the temporomandibular jointcharacteristics based on the 3D model of the patient's face and themarkings of the temporomandibular joint and the condylar position.

Clause 29. The system of clause 21, wherein the 3D data of the patient'sface includes 3D data of the patient's face in with a closed bite andopen lips.

Clause 30. The system of clause 29, further comprising instructions tocapture 2D video of the patient's face as the patient moves their lowerjaw.

Clause 31. The system of clause 30, wherein the instructions to generatean estimate of the temporomandibular joint characteristics based on the3D model of the patient's face include instructions to generate anestimate of the temporomandibular joint characteristics based on the 3Dmodel of the patient's face and the 2D video of the patient's face.

Clause 32. The system of clause 21, wherein the 3D data of the patient'sface includes 3D video of the patient's face.

Clause 33. A system for modeling dental articulation of a patient, themethod comprising: a non-transitory computer readable medium havinginstructions stored thereon that when executed by one or more processorscause the one or more processors to perform a method including:receiving extraoral image data of the patient's dentition in a pluralityof occlusion positions with the extraoral imaging device; receiving a 3Dmodel of the patient's dentition; aligning the 3D model of the patient'sdentition with the patient's dentition in the extraoral images for eachof the plurality of occlusion positions; and modeling dynamic occlusionof upper and lower arches of the patient based on the 3D model of thepatient's dentition aligned with the patient's dentition in theextraoral images for each of the plurality of occlusion positions.

Clause 34. The system of clause 33, wherein the modeling dynamicocclusion is further based on a model of a digital articulator generatedbased on occlusal contacts between an upper jaw and lower jaw of the 3Dmodel of the patient's dentition during simulated movement of the lowerjaw relative to the upper jaw.

Clause 35. The system of clause 33, wherein the method furthercomprises: determining a relationship between lower jaw position and thelocation of external soft tissue of the patient's face; capturing imagedata of the patient chewing with closed lips; and determining theposition of the patient's jaw in the image data of the patient chewingwith closed lips based on the relationship between lower jaw positionand the location of external soft tissue of the patient's face.

Clause 36. The system of clause 35, wherein the image data of thepatient chewing with closed lips is 2D image data.

Clause 37. The system of clause 35, wherein the image data of thepatient chewing with closed lips is 3D image data.

Clause 38. The system of clause 33, wherein the method further comprisessegmenting the patient's dentition in the extraoral image data.

Clause 39. The system of any one of clauses 33-38, wherein the methodfurther comprises extracting 2D dental features from the segmentedextraoral image data of the patient's dentition.

Clause 40. The system of any one of clauses 33-39, wherein the 2D dentalfeatures are one or more of tooth location, tooth outline, gingivalline, and tooth centers.

Clause 41. The system of any one of clauses 33-40, wherein the methodfurther comprises segmenting the 3D model of the patient's dentition.

Clause 42. The system of any one of clauses 33-41, wherein the methodfurther comprises extracting 3D dental features from the segmented 2Dmodel of the patient's dentition.

Clause 43. The system of any one of clauses 33-42, wherein the 3D dentalfeatures are one or more of tooth location, tooth outline, gingivalline, and tooth centers.

Clause 44. The system of any one of clauses 33-43, wherein aligning the3D model of the patient's dentition with the patient's dentition in theextraoral images for each of the plurality of occlusion positionsincludes aligning the 3D dental features with the 2D dental features.

Clause 45. The system of clause 44, wherein the method further comprisesdetermining whether the alignment of the 3D dental features with the 2Ddental features is within a threshold of alignment.

Clause 46. The system of clause 45, wherein the method further comprisesrepeatedly: adjusting the 3D features; attempting to align the 3Dfeatures with the 2D features; and determining whether the alignment ofthe 3D dental features with the 2D dental features is within a thresholdof alignment.

Clause 47. The system of clause 46, wherein adjusting the 3D featuresincludes adjusting a 2D projection of the 3D features.

Clause 48. The system of clause 47, wherein adjusting a 2D projection ofthe 3D features includes adjusting a focal length, virtual cameradistance, or lens distortion of the 2D projection of the 3D features.

Clause 49. The system of any one of the preceding clauses, wherein themethod further comprises interpolating jaw movement between theplurality of occlusion positions.

Clause 50. The system of any one of the preceding clauses, whereininterpolating jaw movement between the plurality of occlusion positionsaccounts for contact between teeth of the upper arch and teeth of thelower arch as the lower jaw moves relative to the upper jaw between theocclusion positions.

Clause 51. The system of any one of the preceding clauses, wherein theocclusion positions include occlusions positions wherein the lower jawis in a neutral bite, a lateral right bite, a lateral left bite, aretraction bite, and a protrusion bite.

Clause 52. The system of clause 51, wherein the occlusion positionsinclude on open bite.

Clause 53. The system of any one of the preceding clauses, wherein theextraoral image data of the patient's dentition are 2D still images ineach of the occlusion positions.

Clause 54. The system of any one of the preceding clauses, wherein theextraoral images of the patient's dentition extraoral image data of thepatient's dentition includes a video as the patient moves their lowerjaw between and to each of the occlusion positions.

Clause 55. The system of any one of the preceding clauses, wherein theextraoral image data of the patient's dentition includes 2D still imagesfrom multiple camera angles in each of the occlusion positions.

Clause 56. The system of any one of the preceding clauses, wherein theextraoral image data of the patient's dentition includes 2D video frommultiple camera angles as the patient moves their lower jaw between andto each of the occlusion positions.

Clause 57. A method for modeling dental articulation of a patient, themethod comprising: capturing 3D data of the patient's face in with theextraoral imaging device; capturing a 3D model of the patient'sdentition; aligning the 3D model of the patient's dentition with 3D dataof the patient's face; and generating an estimate of thetemporomandibular joint characteristics based on the 3D model of thepatient's face; and deriving digital articulator settings for a digitalarticulator based on the 3D model of the patient's dentition with the 3Ddata of the patient's face and the estimate of the temporomandibularjoint characteristics.

Clause 58. The method of clause 57, wherein the 3D data of the patient'sface includes 3D data of the patient's face in with a closed bite andopen lips.

Clause 59. The system of clause 58, wherein the 3D data of the patient'sface includes 3D data of the patient's face in with an open bite andopen lips.

Clause 60. The system of clause 59, wherein the instruction to align the3D model of the patient's dentition with 3D data of the patient's faceincludes aligning registering the 3D model of the patient's dentition tothe 3D data of the patient's face.

Clause 61. The system of any one of clauses 57-60, further comprisinginstructions to generate CBCT data for the internal structure of thepatient's face and jaw.

Clause 62. The system of clause 61, wherein the instructions to generatethe estimate of the temporomandibular joint characteristics based on the3D model of the patient's face further comprise instructions to includeinstructions to generate the estimate of the temporomandibular jointcharacteristics based on the 3D model of the patient's face and the CBCTdata.

Clause 63. The system of clause 57, wherein the instructions includesreceiving markings on the 3D model of the patient's face of thetemporomandibular joint and the condylar position.

Clause 64. The system of clause 63, wherein the instructions to generatethe estimate of the temporomandibular joint characteristics based on the3D model of the patient's face further comprise instructions to includeinstructions to generate the estimate of the temporomandibular jointcharacteristics based on the 3D model of the patient's face and themarkings of the temporomandibular joint and the condylar position.

Clause 65. The system of clause 57, wherein the 3D data of the patient'sface includes 3D data of the patient's face in with a closed bite andopen lips.

Clause 66. The system of clause 65, further comprising instructions tocapture 2D video of the patient's face as the patient moves their lowerjaw.

Clause 67. The system of clause 66, wherein the instructions to generatean estimate of the temporomandibular joint characteristics based on the3D model of the patient's face include instructions to generate anestimate of the temporomandibular joint characteristics based on the 3Dmodel of the patient's face and the 2D video of the patient's face.

Clause 68. The system of clause 57, wherein the 3D data of the patient'sface includes 3D video of the patient's face.

Clause 69. A method comprising: obtaining a first 3D model of an upperjaw of a patient using an intraoral scanner; obtaining a second 3D modelof the lower jaw of the patient using the intraoral scanner; capturing aseries of 2D images of the upper and lower jaws of the patient as thepatient is moves the upper jaw and lower jaw in dynamic occlusion animaging device; processing the captured series of 2D images to identifyfeatures associated with the upper jaw of the patient and the lower jawof the patient; for each 2D image in the captured series of 2D images,identifying a relative position of the first 3D model and the second 3Dmodel based on alignment of features in the first 3D model and second 3Dmodel with the features identified in the 2D image in order to generatea series of relative positions of the first 3D model and the second 3Dmodel; and modeling dynamic occlusion of the upper jaw and the lower jawof the patient based on the series of relative positions of the first 3Dmodel and the second 3D model.

Clause 70. The method of clause 69, wherein the series of 2D imagescomprise near-infrared images.

Clause 71. The method of clause 69, wherein the series of 2D imagescomprise white light images.

Clause 72. The method of clause 69, wherein the series of 2D imagescomprise fluorescence light images.

Clause 73. The method of clause 69, wherein the intraoral scannercomprises multiple cameras for capturing the jaw of the patient fromdifferent angles and wherein capturing the series of 2D images of thejaw of the patient comprises capturing a plurality of 2D images usingthe multiple cameras of the intraoral scanner.

Clause 74. The method of clause 69, wherein the features are anatomicalfeatures.

Clause 75. The method of clause 74, wherein the anatomical features aregingival tissue.

Clause 76. The method of clause 6, wherein the anatomical features areone or more apex of interdental papillia.

Clause 77. The method of clause 74, wherein the anatomical features aretooth surfaces.

Clause 78. The method of clause 74, wherein the features are subsurfacefeatures.

Clause 79. The method of clause 78, wherein the subsurface features areblood vessels.

Clause 80. The method of clause 69, wherein the features are artificialfeatures.

Clause 81. The method of clause 80, wherein the artificial features aretargets affixed to the patient's dentition.

Clause 82. The method of clause 81, wherein the targets are affixed withadhesive or suction.

Clause 83. The method of clause 80, wherein the artificial features aretooth stains.

Clause 84. The method of clause 81, wherein the stains are stainedplaque, caries, or demineralized locations of the teeth.

Clause 85. The method of clause 69, wherein the captured series of 2Dimages includes images captured simultaneously from multiple locations.

Clause 86. The method of clause 85, wherein processing the capturedseries of 2D images to identify features includes processing the imagessimultaneously from multiple locations.

Clause 87. The method of clause 69, wherein capturing the series of 2Dimages comprises capturing images using multiple modalities and whereinthe features are identified and their locations determined in thedifferent modalities.

Clause 88. The method of clause 69, wherein the captured series of 2Dimages to identify features includes captured images of repeated motionof the patient's teeth and wherein processing, includes determining anaverage trajectory of motion of the patient's teeth based on determinedlocations of the patient's jaws.

Clause 89. The method of clause 69, wherein the imaging device is anextraoral imaging device.

Clause 90. The method of clause 69, wherein the imaging device is anintraoral scanner.

Clause 91. The method of clause 69, wherein processing includes: (a)determining a camera position of the 2D image; (b) determine positionsof the teeth of the upper and lower jaws based on the camera position;(c) determining a difference in positions of the teeth of the upper andlower jaws based on the camera position to positions of teeth in a 3Dmodel of the upper and lower jaws; (d) updating the camera position ofthe 2D model; and (e) repeating (a) through (d) until the difference isless than a threshold.

Clause 92. A system comprising: a processor; and non-transitory computerreadable medium comprising instructions that when executed by theprocessor cause the system to carry out the method of any one of clauses69-91.

Embodiments of the present disclosure have been shown and described asset forth herein and are provided by way of example only. One ofordinary skill in the art will recognize numerous adaptations, changes,variations and substitutions without departing from the scope of thepresent disclosure. Several alternatives and combinations of theembodiments disclosed herein may be utilized without departing from thescope of the present disclosure and the inventions disclosed herein.Therefore, the scope of the presently disclosed inventions shall bedefined solely by the scope of the appended claims and the equivalentsthereof.

What is claimed is:
 1. A system for modeling dental articulation of apatient, the method comprising: a non-transitory computer readablemedium having instructions stored thereon that when executed by one ormore processors cause the one or more processors to perform a methodincluding: receiving 2D image data of the patient's dentition in aplurality of occlusion positions; receiving a 3D model of the patient'sdentition; aligning the 3D model of the patient's dentition with thepatient's dentition in the extraoral images for each of the plurality ofocclusion positions; and modeling dynamic occlusion of upper and lowerarches of the patient based on the 3D model of the patient's dentitionaligned with the patient's dentition in the extraoral images for each ofthe plurality of occlusion positions.
 2. The system of claim 1, whereinmodeling dynamic occlusion is further based on a model of a digitalarticulator generated based on occlusal contacts between an upper jawand lower jaw of the 3D model of the patient's dentition duringsimulated movement of the lower jaw relative to the upper jaw.
 3. Thesystem of claim 1, wherein the method further comprises: determining arelationship between lower jaw position and the location of externalsoft tissue of the patient's face; capturing image data of the patientchewing with closed lips; and determining the position of the patient'sjaw in the image data of the patient chewing with closed lips based onthe relationship between lower jaw position and the location of externalsoft tissue of the patient's face.
 4. The system of claim 3, wherein theimage data of the patient chewing with closed lips is 2D image data. 5.The system of claim 3, wherein the image data of the patient chewingwith closed lips is 3D image data.
 6. The system of claim 1, wherein themethod further comprises segmenting the patient's dentition in theextraoral image data.
 7. The system of claim 6, wherein the methodfurther comprises extracting 2D dental features from the segmentedextraoral image data of the patient's dentition.
 8. The system of claim7, wherein the 2D dental features are one or more of tooth location,tooth outline, gingival line, and tooth centers.
 9. The system of claim8, wherein the method further comprises segmenting the 3D model of thepatient's dentition.
 10. The system of claim 9, wherein the methodfurther comprises extracting 3D dental features from the segmented 2Dmodel of the patient's dentition.
 11. The system of claim 10, whereinthe 3D dental features are one or more of tooth location, tooth outline,gingival line, and tooth centers.
 12. The system of claim 11, whereinaligning the 3D model of the patient's dentition with the patient'sdentition in the extraoral images for each of the plurality of occlusionpositions includes aligning the 3D dental features with the 2D dentalfeatures.
 13. The system of claim 12, wherein the method furthercomprises determining whether the alignment of the 3D dental featureswith the 2D dental features is within a threshold of alignment.
 14. Thesystem of claim 13, wherein the method further comprises repeatedly:adjusting the 3D features; attempting to align the 3D features with the2D features; and determining whether the alignment of the 3D dentalfeatures with the 2D dental features is within a threshold of alignment.15. The system of claim 14, wherein adjusting the 3D features includesadjusting a 2D projection of the 3D features.
 16. The system of claim15, wherein adjusting a 2D projection of the 3D features includesadjusting a focal length, virtual camera distance, or lens distortion ofthe 2D projection of the 3D features.
 17. The system of claim 16,wherein the method further comprises interpolating jaw movement betweenthe plurality of occlusion positions.
 18. The system of claim 17,wherein interpolating jaw movement between the plurality of occlusionpositions accounts for contact between teeth of the upper arch and teethof the lower arch as the lower jaw moves relative to the upper jawbetween the occlusion positions.
 19. The system of claim 1, wherein the2D image data is captured by an intraoral scanner and wherein the 3Ddata is captured by the intraoral scanner.
 20. The system of claim 1,wherein the 2D image data is captured by an extraoral imaging device.